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ABSTRACT 


Marine  Corps  Manpower  Plans  and  Policy  Division,  Manpower  &  Reserve 
Affairs,  is  responsible  for  formulating  Marine  Corps  force  manpower  plans. 
Accomplishing  this  mission  requires  extensive  knowledge  of  the  Human 
Resource  Development  Process  (HRDP)  for  controlling  future  personnel  attrition, 
retention,  and  accession  quantities  to  ensure  appropriate  quantities  of  its  various 
Military  Occupational  Specialties  (MOS)  and  overall  end  strength  are  maintained. 
To  assist  their  mission,  an  agent-based  computer  simulation  model  was 
developed  in  the  Java  computer  language.  This  thesis  investigates  that 
simulation  model,  titled  Manpower  Simulation  Model  (MSM).  This  thesis  provides 
documentation  of  MSM’s  architecture  and  processes,  tests  the  sensitivity  of  its 
inputs  through  the  use  of  an  experimental  design,  and  validates  MSM’s  output 
measures  by  calculating  the  relative  error  for  five  successive  forecast  years  for 
various  HRDP  categories.  This  thesis  found  that  MSM’s  structure  and  output 
measurement  responses  aligned  with  HRDP  practices.  With  respect  to  validation, 
on  average  the  HRDP  categories  losses  and  accessions  underestimated  by  17 
and  18  percent,  respectively,  while  gains  overestimated  by  36  percent.  The 
category  promotions  generally  underestimated,  but  lessened  in  magnitude  as 
grade  increased.  The  category  retention  consistently  overestimated  for  all 
grades.  Lastly,  the  MSM  showed  biasness  toward  retaining  Marines  over 
backfilling  vacancies  through  accessions. 
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I.  INTRODUCTION 


A.  BACKGROUND 

The  Marine  Corps  is  regarded  as  being  the  Nation’s  premier  expeditionary 
force  in  readiness.  Before  it  can  ever  be  deployed  to  conduct  its  various  ranges 
of  military  operations,  it  must  ensure  it  has  the  proper  personnel  in  place  to 
execute  those  missions.  Annotated  as  the  Human  Resources  Development 
Process  (HRDP),  the  Marines  and  civilians  at  Manpower  and  Reserve  Affairs 
(M&RA)  wrestle  with  this  convoluted  and  mission  critical  process  to  ensure  the 
Marine  Corps  has  the  proper  number  of  Marines  proportionately  allocated  across 
its  various  Military  Occupations  Specialties  (MOS).  To  add  an  additional  layer  of 
difficultly,  M&RA  must  accomplish  this  mission  for  an  organization  that  has  an 
extreme  flux  in  personnel  change. 

Averaging  from  the  past  decade,  each  year  there  are  approximately 
30,000  Marines  that  choose  to  enter  and  exit  the  Marine  Corps  (Spafford,  2016). 
Roughly  60%  of  Marines  are  on  their  first  enlistment  contract,  45%  are  below  the 
paygrade  of  E-4,  and  75%  will  not  reenlist  for  a  second  contract  (Spafford,  2016). 
Figure  1  shows  this  dramatic  manpower  drop  of  Enlisted  Marines  exiting  the 
Marine  Corps  after  their  first  enlistment  contract.  This  significant  drop  is  due  to 
the  general  manpower  structure  of  the  Marine  Corps,  as  well  as  its  merit-based 
promotion  selection  that  is  in  conjunction  with  the  “up  or  out”  manpower  system. 
To  ensure  mission  accomplishment  within  M&RA,  it  is  crucial  to  understand 
these  factors  that  drive  manpower  movements.  The  manpower  process  is 
expanded  on  in  Section  C  or  this  chapter. 
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Year 


Figure  1.  Marine  Corps  Enlisted  Force  Fiscal  Year  (FY)  1990-2000. 

Source:  Rostker  (2013). 

Advances  in  technology  have  facilitated  the  ability  to  develop  tools  that 
yield  enhanced  manpower  analysis.  Flowever,  most  of  these  decision  making 
tools  are  still  relatively  isolated  models  that  provide  insightful  information  in  a 
narrow  spectrum  of  the  HRDP  (R.  A.  Garrick,  personal  communication,  April  6, 
2016).  Naturally,  policy  and  procedures  implemented  within  M&RA  try  to  mitigate 
the  unintended  consequences  and  errors  of  assembling  the  various  patches  of 
information.  To  bridge  this  gap,  the  power  of  computer  simulation  was  explored 
with  the  intent  to  integrate  the  numerous  processes  of  HRDP  into  a  single 
simulation  model. 

The  first  documented  computer  modeling  software  that  attempted  to 
provide  a  HRDP  whole-system  oriented  viewpoint  was  an  Arena  model.  Created 
by  Rockwell  Automation,  Arena  is  a  simulation  software  that  models  business 
practices  to  assist  that  business  in  its  “ability  to  analyze  and  make  decisions  on 
how  to  improve  (their)  process”  (Rockwell  Automation,  2016).  Used  at  M&RA 

from  roughly  2008  to  2011,  this  model  was  cumbersome,  slow,  and  complex. 
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Ultimately,  its  inability  to  forecast  numerous  years  into  the  future  and 
undocumented  source  code  methods  led  this  software  program  to  become 
untrusted,  impractical,  and  unused  to  the  point  that  its  contracting  license  with 
Rockwell  Automation  was  not  renewed  upon  its  licensing  expiration  (M.  Ramirez, 
personal  communication,  April  4,  2016). 

In  2011,  former  Naval  Postgraduate  School  Operations  Research 
graduate  Captain  Ronald  Garrick,  USMC  (ret),  arrived  at  Manpower  Studies  and 
Analysis  Branch,  M&RA,  thereupon  noticing  the  technology  capability  gap  and 
began  developing  what  ultimately  became  the  Manpower  Simulation  Model 
(MSM). 

The  MSM  is  a  Java  language  agent-based  simulation  program  that  aims 
to  integrate  the  numerous  processes  of  the  HRDP  under  a  single  computer 
simulation  model.  Using  the  Marine  Corps’  force  structure  as  its  virtual 
architecture  and  manpower  shaping  rules  as  its  internal  constraints,  the  program 
combines  the  deterministic  aspects  of  the  HRDP  with  a  stochastic  simulation 
capability  that  produces  a  multitude  of  output  metrics.  These  output  metrics  aid 
the  decision  makers  at  M&RA  to  gain  insightful  forecasted  metrics  that  range 
across  numerous  fiscal  years,  and  breaks  down  into  the  dimensional  analysis  at 
the  MOS  skill  and  grade  level. 

B.  PURPOSE 

This  thesis  provides  an  overall  evaluation  of  the  MSM  and  its  potential  for 
future  usage  so  that  its  output  metrics  can  be  used  with  trust  and  confidence  in 
making  manpower  decisions  by  the  leaders  within  M&RA.  This  was 
accomplished  by  inspecting  and  documenting  the  various  internal  structures  and 
procedures  within  the  MSM.  A  software-verification  approach  was  taken  to  test 
the  sensitivity  of  its  input  metrics  through  the  use  of  an  experimental  design. 
Finally,  a  software-validation  approach  was  used  to  evaluate  the  model’s 
accuracy  against  historical  Marine  Corps  manpower  data. 


3 


C.  THE  MANPOWER  PROCESS 

To  understand  the  underlying  concepts  of  the  MSM,  and  its  goal  of 
providing  forecasted  metrics,  one  first  has  to  understand  the  enlisted  HRDP.  The 
intention  of  this  section  is  to  provide  the  reader  with  an  overview  of  this  process, 
rather  than  an  in-depth  detailed  analysis,  to  serve  as  the  basis  for  comparison  to 
the  MSM.  This  section  will  highlight  the  conjoined  process  for  Officer  and 
Enlisted  personnel  where  appropriate.  However,  the  scope  of  this  thesis  is  purely 
focused  on  enlisted  personnel,  since  that  is  the  personnel  parameter  used  within 
the  MSM. 

Forecasting  manpower  quantities  are  derived  from  two  critical 
components;  the  absolute  manpower  structure  of  the  Marine  Corps  and  the 
personnel  movement  that  occurs  within  it.  The  first  part  of  this  section  will  focus 
on  manpower  structure.  Comprising  of  approximately  240  MOSs  that  are  spread 
throughout  9  Enlisted  grades,  the  Marine  Corps  Enlisted  Force  averages  around 
162,000  Marines  at  any  given  time  (Manpower  Plans  and  Policy  Division,  2016). 

Determining  what  the  force  structure  should  be  for  a  given  fiscal  year  is  a 
cumbersome  process.  The  enlisted  force  is  derived  from  manpower  requirements 
or  previous  force  structures.  These  requirements  are  determined  from 
occupational  field  (occField)  sponsors  who  are  specialized  and  advocate  for 
occFields  to  possess  specific  levels  of  experience,  where  experience  can  be 
synonymous  with  a  Marine’s  time  in  service,  that  are  proportionality  distributed 
across  the  Marine  Corps  in  terms  of  both  depth  and  breadth.  The  sponsor 
receives  guidance  from  the  Commandant  of  the  Marine  Corps  in  his  strategic 
analysis  of  various  documents  such  as  the  National  Security  Strategy  and  the 
National  Defense  Strategy,  as  well  as  from  current  Combatant  Commanders  who 
yield  bottom-up  refinement  to  previously  established  force  structures  (United 
States  Marine  Corps,  2009).  The  Deputy  Commandant  Combat  Development 
and  Integration  (DC  CD&I)  ultimately  approves  the  new  requirements  list,  which 
appropriately  becomes  the  documented  Table  of  Organization  (T/O).  Figure  2 
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depicts  this  process,  as  described  in  Marine  Corps  Order  (MCO)  531 1.1  D  Total 
Force  Structure  Process. 
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Figure  2.  The  Marine  Corps  Total  Force  Structure  Process.  Adapted  from 

United  States  Marine  Corps  (2009). 


The  T/O  is  therefore  the  foundational  document  toward  which  the  Marine 
Corps  tries  to  build  and  sustain  its  force,  which  is  accomplished  by  “buying  billet 
seats.”  Every  purchased  billet  has  political  and  budgetary  constraints  associated 
with  it.  The  process  of  “buying  billet  seats”  is  synonymous  with  the  fiscal  year’s 
approved  end  strength  as  provided  by  the  National  Defense  Authorization  Act 
(U.S.  Department  of  Defense,  1996).  When  differences  between  what  the  Marine 
Corps  would  like  to  have,  the  T/O,  and  what  the  Marine  Corps  is  allowed  to  have, 
its  approved  end  strength,  the  Manning  and  Staffing  Precedence  Order  is  taken 
into  account.  The  Manning  and  Staffing  Precedence  Order  is  a  MCO  that 
identifies  which  units  will  receive  priority  in  fulfilling  billet  vacancies,  as  directed 
by  the  Commandant  of  the  Marine  Corps  (United  States  Marine  Corps,  2012). 
The  combination  of  these  steps  in  addition  to  the  original  T/O  produces  the 
Authorized  Strength  Report  (ASR). 

The  ASR  is  the  principal  document  that  M&RA  receives  to  complete  the 
final  components  of  the  HRDP.  However,  before  it  can  be  used  it  has  to  be 
reconciled  to  be  compliant  with  political  and  budgetary  constraints.  For  budgetary 
constraints,  the  grade  quantities  are  aggregated  and  summed  with  respect  to  that 

grade’s  pay  to  ensure  that  monetary  personnel  limits  are  not  exceeded. 
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Within  Manpower  Plans  and  Policy  Division  (MP),  additional  manning 
controls  are  implemented  to  account  for  the  Marine  Corps’  patients,  prisoners, 
transients,  and  trainees  (P2T2).  Accounting  for  P2T2  against  the  ASR  produces 
the  Grade  Adjusted  Recapitulation  (GAR).  The  manpower  metrics  specified  in 
the  GAR  serve  as  the  final  manpower  targets  for  a  specific  fiscal  year.  A 
snapshot  of  the  GARs  manpower  metrics  may  be  viewed  in  Appendix  B.  The 
GAR  is  the  manpower  structure  document  the  MSM  uses  to  build  its  virtual 
architecture  from. 

Building  off  of  the  absolute  structure,  personnel  movement  becomes  the 
second  critical  component  to  properly  forecasting  manpower  requirements.  This 
component  is  often  illustrated  in  a  graphic  called  the  Enlisted  Manpower 
Pyramid,  shown  in  Figure  3.  It  includes  external  and  internal  personnel 
movement,  which  is  classified  into  four  categorical  areas:  accessions/gains, 
retentions/reenlistments,  promotions,  and  losses. 


Figure  3.  Enlisted  Manpower  Pyramid.  Adapted  from  Spafford  (2016). 
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These  four  components  yield  insight  into  how  Marines  are  moving  through 
the  manpower  structure.  These  four  components  are  further  broken  down  to 
capture  specific  manpower  movement.  For  accessions/gains,  sub-categories 
include  Marines  returning  from: 

•  Medical  Holds 

•  End  of  Active  Service  (EAS)  Holds 

•  Deserter  Status 

•  Fulfilling  Recruiter  Billets 

•  Cross  Year  Extensions 

•  Non-Prior  Service  Accessions 

For  retentions/reenlistments,  this  category  is  broken  down  into  two  sub¬ 
categories:  First  Term  Alignment  Plan  (FTAP)  and  the  Subsequent  Term 
Alignment  Plan  (STAP).  These  plans  shape  the  inventory  of  the  enlisted  force  by 
controlling  the  number  of  enlistments  for  the  respective  term  force  (United  States 
Marine  Corps,  2010).  These  plans  are  critical  as  they  ensure  the  proper  quantity 
of  MOSs,  with  respect  to  grade  distribution,  are  being  retained  within  the  Marine 
Corps. 

The  component  promotions  does  not  have  a  specific  sub-category, 
however  it  is  proportionally  controlled  by  the  Enlisted  Career  Force  Controls 
(ECFC).  ECFC  impacts  two  specific  areas,  first  they  impact  the  promotion  zones 
control  parameters.  These  controls  ensure  each  Marine  receives  an  opportunity 
to  be  promoted,  as  well  as  ensures  a  Marine  has  the  proper  experience  to  fulfill 
higher  grade  responsibilities.  Secondly,  the  controls  facilitate  the  process  of  only 
retaining  the  most  qualified  Marines  by  forcing  Marines  who  fall  below  acceptable 
career  progression  standards  out  of  the  service.  For  example,  if  Marines  are 
continuously  being  passed  over  for  promotion  it  is  assumed  they  do  not  possess 
the  high-standard  qualifications  the  Marine  Corps  is  looking  for  and  therefore 
should  not  continue  to  be  retained  within  the  force  (United  States  Marine  Corps, 
2010). 
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The  component  losses  has  multiple  sub-categories,  specifically 
categorizing  Marines  that: 

•  Receive  commission  and  become  Officers,  referred  to  as  an 
enlisted  to  officer  (E20)  transition 

•  Drop  out  and  separate  from  Boot  Camp 

•  Transition  into  the  reserves 

•  Reach  service  limitations 

•  Retire 

•  Execute  general  EAS 

•  Death  cases 

•  Non-EAS  cases 

Aggregating  and  associating  the  four  manpower  movement  categories 
yields  detailed  rates  for  Marines  moving  among  the  manpower  structure. 
Cognizance  of  these  rates  allow  manpower  planners  to  take  today’s  force, 
extrapolate  the  data,  and  forecast  what  they  believe  will  be  the  future’s  force. 
Projections  are  developed  at  the  beginning  of  each  fiscal  year  to  estimate  the 
amount  of  Marines  moving  within  the  manpower  pyramid  in  order  to  anticipate 
the  quantity  of  accessions  needed  to  backfill  future  vacancies.  These  forecasts 
allow  the  Marine  Corps  to  establish  milestones  that  facilitate  it  to  conclude  a 
fiscal  year  exactly  at  the  National  Defense  Authorization  Act’s  approved  end 
strength.  This  is  the  manpower  management  game  Manpower  Plans  and  Policy 
Division  is  continuously  playing,  and  its  success  is  crucial  to  ensure  the  Marine 
Corps’  manpower  readiness. 
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II.  TECHNICAL  REVIEW 


This  chapter  serves  to  supply  the  reader  with  the  technical  background 
knowledge  that  will  facilitate  the  understanding  of  MSM’s  construction.  This  is  not 
an  exhaustive  background  but  rather  focuses  on  the  key  components  that  give 
the  MSM  its  greatest  capabilities.  Finally,  this  chapter  will  conclude  with  an 
overview  of  the  scientific  methodologies  used  for  the  evaluation  of  the  MSM’s 
input  and  processes  sensitivity. 

A.  AGENT-BASED  MODELING  AND  SIMULATION 

Prior  to  agent-based  simulation  modeling,  computer  models  used 
mathematical  formulas  to  compute  statistics  of  interest  (Railsback  &  Grimm, 
2012).  As  object  oriented  computer  languages  have  impacted  computer 
programming,  its  methodology  has  also  impacted  modeling  and  the  ability  to 
track  objects,  annotated  as  agents  within  simulation,  through  the  system. 

“Agent-based  modeling  are  thus  models  where  individual  agents  are 
described  as  unique  and  autonomous  entities  that  usually  interact  with  each 
other  and  their  environment  locally”  (Railsback  &  Grimm,  2012,  p.  10).  Agents 
are  entities  that  can  be  constructed  to  have  very  specific  behaviors.  As  entities, 
they  are  able  to  retain  the  statistics  of  their  interaction  with  the  system,  which 
allows  for  enhanced  data  analysis.  This  includes  analyzing  how  the  agent  is 
impacted  by  the  rules  and  processes  within  the  system  (Railsback  &  Grimm, 
2012). 

This  type  of  problem  framing  allows  for  three  separate  aspects  to  be 
examined:  the  agent  as  an  individual,  agents  aggregated  to  form  a  group,  and 
the  system  as  a  whole.  These  aspects  facilitate  the  user’s  ability  to  examine  the 
model  and  the  total  throughput  of  an  agent  within  a  system,  and  more  importantly 
gain  a  further  understanding  of  the  third,  fourth,  even  tenth-order  consequences 
associated  with  an  environment  policy  decision  (Railsback  &  Grimm,  2012). 
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B.  STATES,  EVENTS,  AND  EVENT  GRAPH  METHODOLOGY 

In  modeling  and  simulation,  understanding  the  throughput  of  the 
numerous  processes  and  variables  that  are  adjusted  within  the  simulation  can  be 
a  daunting  task.  To  simplify  these  actions  and  understand  the  numerous 
calculations  that  occur,  these  processes  can  be  broken  down  into  a  few  graphic 
illustrations.  For  a  majority  of  simulations,  to  include  the  MSM,  the  two 
predominant  graphic  illustrations  are  state  variables  and  events. 

A  state  variable  is  a  variable  that  “has  a  possibility  of  changing  value  at 
least  once  during  any  given  simulation  run”  (Buss,  2011,  p.  1-1).  These  items 
provide  the  user  with  a  snapshot  description  of  the  system  at  any  specific  point  in 
time.  Over  the  course  of  a  simulation,  state  variables  are  identified  as  being 
piecewise  constants.  This  means  that  a  state  variable  can  only  change  its  value 
instantaneously  at  a  specified  trigger  point.  These  trigger  points  can  be 
continuous  time  steps,  which  is  the  method  of  advancement  for  the  MSM,  or  at 
previously  specified  non-continuous  discrete  times.  This  capability  for  a  state  to 
change,  to  transition  to  a  new  state,  introduces  the  constraint  called  an  event 
(Buss,  2011). 

An  event  is  a  particular  action  that  occurs  within  a  simulation,  and  its 
occurrence  is  the  trigger  for  a  state  variable  to  change  its  value  instantaneously. 
Events  can  be  mapped  to  change  a  plethora  of  state  variables  in  a  singular 
instance,  or  as  little  as  none.  The  key  take  away  is  that  no  state  variable  is 
allowed  to  change  its  value  without  being  initiated  to  do  so  by  the  instance  of  an 
event  (Buss,  2011). 

Various  practices  have  been  developed  to  illustrate  these  state  variables 
and  event  interactions;  one  such  practice  used  is  the  event  graph.  An  event 
graph  is  an  illustration  that  depicts  states,  the  possible  transitions  that  a  state  can 
execute,  the  state  variables  that  are  required  to  be  updated  with  the  execution  of 
that  event,  and  any  constraints  that  are  emplaced  for  the  event  to  occur.  The 
aggregation  of  these  illustrations  describes  a  particular  process  within  the  model; 
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and  is  captured  by  an  event  graph  (Buss,  2011).  Figure  4  is  an  event  graph  that 
depicts  a  specific  process;  in  the  case  of  this  example  a  Multiple  Server  Queue 
process.  The  figure  is  annotated  with  numerous  descriptions  to  identify  the 
various  components  of  an  event  graph.  Dotted  red  arrows  link  these  annotations 
to  their  respective  event  graph  illustrations. 


Event  Node  Edge  Transition 


Figure  4.  Multiple  Server  Queue  Event  Graph  with  Legend. 

Adapted  from  Buss  (201 1 ). 


Within  an  event  graph,  each  event  is  represented  by  its  annotated  circle, 
called  a  node.  Directly  under  the  node,  the  states  that  are  to  be  updated  by  the 
execution  of  that  particular  event  are  represented  in  equation  form.  Connecting 
nodes  to  each  other  is  accomplished  by  a  one-way  arrow,  commonly  referred  to 
as  an  edge.  Because  this  edge  implicates  a  transition  that  can  only  occur  in  one 
specific  direction,  this  graph  is  a  directed  graph.  Resting  in  the  middle  of  the 
edges  are  the  conditional  statements  that  serve  as  requirements  that  constrain 
the  next  possible  event  from  occurring.  These  conditions  are  represented  in 
Boolean  form;  they  are  either  true  and  allow  the  transition  to  occur  or  they  are 
false  and  disallow  it.  Located  at  the  beginning  of  each  edge  are  any  parameters 

and  arguments  that  are  passed  to  the  next  event  to  specify  how  it  will  be 
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executed.  These  parameters  and  arguments  can  serve  as  metric  references  that 
affect  how  certain  state  functions  are  executed,  or  they  can  serve  as  dimensional 
parameters  that  are  used  to  reference  specific  states  in  the  state  transition 
formulas  (Buss,  2011). 

Within  the  context  of  the  MSM,  event  graphs  become  extremely  useful  in 
understanding  its  numerous  processes  executed.  Specifically,  they  facilitate  the 
ability  to  easily  view  the  process’s  logic,  and  compare  it  to  actual  Marine  Corps 
practices.  Chapter  III  documents  the  structure  and  main  processes  within  the 
MSM.  Specific  MSM  event  graphs  can  be  found  there. 

C.  RANDOM  NUMBER  GENERATORS 

The  Marine  Corps  manpower  throughput  process,  the  HRDP,  is 
fundamentally  built  off  of  the  individual  Marine.  As  such,  while  the  HRDP  is  a 
discrete  process  it  naturally  must  account  for  the  human  psychological  factors 
that  influence  every  Marine.  The  Marine  Corp’s  Doctrinal  Publication  1, 
Warfighting,  identifies  human  factors  as  the  intangible  dimension  of  warfare  and 
is  the  root  cause  of  “complexities,  inconsistencies,  and  peculiarities”  that  shape 
war  (MCDP-1,  1997).  With  respect  to  manpower,  this  is  most  notably  their  free 
will  and  the  affordability  to  make  decisions  that  affect  their  career.  These  human 
factors  can  influence  a  Marine’s  decision  to  reenlist  or  execute  an  EAS 
discharge,  continue  within  their  MOS  or  lateral  move  into  a  new  MOS,  or  in  the 
rare  case  decide  to  desert  the  service.  Human  factors  are  also  present  at  other 
key  HRDP  events.  Take  for  example  a  promotion  board,  while  the  general  intent 
is  to  promote  the  most  qualified  Marines,  the  promotion  selection  is  always 
subject  to  the  human  factors  of  the  board  and  their  collaborative  decision  making 
process.  One  method  to  account  for  the  selection  and  execution  of  non- 
deterministic  events  is  to  incorporate  randomness  into  a  simulation  model 
(Sanchez,  2007). 

Within  the  MSM,  a  Mersenne  Twister  random  number  generator  is  used  to 
invoke  instances  of  randomness  during  the  HRDP.  Once  invoked,  the  randomly 
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generated  number  is  compared  to  an  event  likelihood,  which  is  referenced  from  a 
database,  to  determine  if  the  event’s  hypothesis  is  executed  (Garrick,  2014).  This 
usage  of  a  random  number  generator  gives  the  MSM  its  stochastic  modeling 
feature.  Stochastic,  in  terms  of  modeling  and  simulation,  can  be  defined  as  when 
“the  same  set  of  simulation  inputs  may  produce  different  outputs”  (Sanchez  & 
Wan,  2012,  p.  2). 

The  Mersenne  Twister  was  invented  in  1997  by  Makoto  Matsumoto  and 
Takuji  Nishimura.  It  employs  219937"1  iterative  possibilities  that  spread  across  623 
dimensions,  when  using  a  32-bit  computer  operating  system  (Tia  &  Benkrid, 
2009).  The  Mersenne  Twister  is  an  incredibly  effective  random  number  generator 
and  has  become  the  most  widely  used  random  number  generator  in  modeling 
and  simulation  (Tia  &  Benkrid,  2009). 

Stochastic  Modeling  provides  additional  analytic  capabilities  to  the  user. 
Unlike  deterministic  models  that  yield  consistent  point  output  measurements,  a 
stochastic  model  will  distribute  those  measurements  over  a  range.  This 
distribution  allows  for  the  mean  and  standard  deviation  of  an  output 
measurement  to  be  calculated,  which  in  turn  can  be  derived  into  likelihood. 
However,  in  order  to  produce  an  effective  distribution  of  output  measurements 
the  model  must  be  repeated  multiple  times  (Law  &  Kelton,  2000).  Further 
information  regarding  the  implementation  and  usage  of  random  number 
generators  within  the  MSM  can  be  found  in  Chapter  III. 

D.  DESIGN  OF  EXPERIMENTS 

Design  of  Experiments  (DOE)  is  a  statistical  approach  in  conducting 
experiments  to  yield  information  about  the  processes  of  a  model  and  how  its 
outputs  respond  to  changes  in  the  inputs.  The  information  produced  from  these 
experiments  can  identify  minimum  and  maximum  thresholds  for  inputs,  capture 
how  inputs  drive  the  model  to  produce  its  output  measurements,  as  well  as 
understand  how  policies,  codified  as  processes  within  the  model,  affect  the 
model’s  output  measurements  (Sanchez  &  Wan,  2012).  History  has  naturally 
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utilized  DOE  within  the  agriculture  industry;  seen  countless  times  in  the  simple 
scenario  of  a  farmer  trying  two  different  fertilizes  on  his  crops  to  identify  which 
produces  the  best  crop  yield.  With  the  aid  of  computer  simulation,  DOE’s 
capabilities  have  expanded  to  examine  and  evaluate  complex  system  models 
before  they  are  ever  implemented  into  the  physical  world  (Sanchez  &  Wan, 
2012). 

Specific  terminology  and  nomenclature  are  used  to  codify  critical  parts  and 
processes  of  a  DOE.  The  inputs  of  a  system  that  are  to  be  strategically  altered 
per  simulation  trial  are  called  factors.  The  number  of  values  a  factor  can  be 
simulated  at,  also  referred  as  the  discrete  range  of  values  for  a  specific  factor,  is 
annotated  as  levels.  The  interaction  between  the  varying  input  factors  and  the 
model’s  output  measures  is  annotated  as  the  model’s  response  surface.  Lastly,  a 
trial  instance  of  the  various  factors  set  at  specific  levels  is  called  a  design  point 
(Sanchez  &  Wan,  2012). 


Response  surfaces  can  be  expressed  mathematically,  which  in  turn  can 
be  shown  as  a  graphical  illustration.  These  expressions  are  called  metamodels 
and  link  how  the  various  input  factors  impact  the  model’s  output  measures.  How 
input  factors  effect  a  model’s  response  surface  can  be  measured  by  observing 
two  different  impact  methods:  those  that  show  solely  main  effects  and  those  that 
show  main  effects  with  additional  quadratic  effects  (Sanchez  &  Wan,  2012).  Main 
effects  are  regarded  as  factors  that  independently  impact  the  models  outputs. 
Specifically,  these  are  the  effects  captured  when  a  singular  factor  is  altered. 
Quadratic  effects  reveal  how  factors  impact  themselves  quadratically,  as  well  as 
how  factors  interact  with  other  factors  in  two-way  interactions.  Equations  (1)  and 
(2)  show  the  formula  representation  for  main-effects  and  quadratic-effects, 
respectively  (Sanchez  &  Wan,  2012). 


Y  +£ 

i= 1 


(1) 
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r  =  A  + 1 A  x, ,  + 1 ( x,  -  x, .f  +  g  t  A. ■  ( x,  ~  x, )  ( >  -  x , )  +  * 

i= 1  >=l  i=l  >1+1  (2) 

The  final  aspect  to  DOE  is  the  design  itself,  which  defines  how  the  factors 
will  be  altered  per  design  point  for  the  entirety  of  the  DOE  experiment.  By  varying 
these  factors  in  a  strategic  way  the  design  can  simulate  the  model  over  the  range 
of  possible  inputs,  called  the  sample  space  (Sanchez  &  Wan,  2012). 

Using  DOE’s  previous  employment  in  agriculture  to  illustrate  this  idea  of  a 
design,  let  us  create  the  case  study  to  examine  under  what  conditions  will 
produce  the  greatest  crop  yield  for  a  farmer.  In  this  experiment,  three  factors  are 
chosen  to  be  varied;  seed  type,  fertilizer  type,  and  pesticide  type.  Each  factor  has 
two  types  of  brands  that  will  be  used;  these  are  the  levels  for  those  factors. 
Listing  out  every  possible  configuration  yields  eight  design  points  for  the  DOE 
experiment.  This  type  of  design  is  called  a  full  factorial  design  with  three  factors 
altered  at  two  levels;  it  is  annotated  as  23.  Designs  can  be  illustrated  graphically 
or  annotated  in  matrix  form  to  identify  the  range  of  possibilities,  the  sample 
space,  that  the  factors  of  a  DOE  can  assume  (Sanchez  &  Wan,  2012).  Figure  5 
shows  a  23  factorial  design  in  matrix  and  graphical  form.  Graphical  images  are 
great  for  viewing  the  sample  space  and  observing  how  the  design  points  fill  it. 
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Figure  5.  23  Factorial  design,  graphically  and  in  matrix  form,  with 
numbered  design  points.  Source:  Sanchez  &  Wan  (2012). 
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Filling  the  sample  space  with  design  points  introduces  the  fundamental 
obstacle  that  must  be  overcome  with  DOE  designs.  On  one  side  of  the  problem, 
a  user  wants  to  obtain  detailed  information  of  how  a  model  responds  to  its  inputs. 
However,  to  obtain  a  perfect  model  synopsis  a  user  would  need  an  infinite 
number  of  design  points,  which  the  computational  costs  to  calculate  would  be 
unfathomable.  Conversely,  to  achieve  a  low  computational  cost  model  a  user 
might  be  sacrificing  the  level  of  detail  the  model’s  response  surface  yields. 

To  illustrate  this  problem  better,  let’s  use  the  children’s  game  capture  the 
flag.  In  this  game,  there  are  two  factors  that  can  be  used  to  evaluate  how  good  a 
child  is  at  playing  the  game;  they  are  stealth  and  speed.  If  the  factors  are 
restricted  to  only  two  levels,  the  sample  space  is  left  fairly  open  and  a  child  can 
only  be  categorized  into  extreme  stealth-speed  options.  However,  when 
additionally  levels  are  examined  per  factor  the  experimental  space  is  filled  with 
more  stealth-speed  possibilities.  This  leaves  the  user  with  more  possibilities  to 
classify  players,  and  therefore  better  identify  the  specific  factor  levels  where  the 
best  players  resides  (Sanchez  &  Wan,  2012).  This  effect  is  illustrated  in  Figure  6. 
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Figure  6.  22and  ll2  Factorial  Designs  for  Capture-the-Flag.  Source: 

Sanchez  &  Wan  (2012). 

As  seen  in  Figure  6,  being  able  to  better  fill  the  design’s  experimental 
space  with  design  points  yields  substantially  more  information  about  a  system. 
This  technique  is  called  space-filling  (Sanchez  &  Wan,  2012).  However,  while  it 
provides  detailed  information  of  a  model’s  response  surface  it  is  extremely 
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computationally  expensive  to  run  through  all  of  the  specific  calculations. 
Comparing  Figure  6’s  two  DOE  designs,  the  22  factorial  design  requires  four 
simulation  runs  where  the  ll2  factorial  design  requires  121  simulation  runs.  One 
solution  to  lower  computational  costs  while  acquiring  detailed  information  of  a 
model  is  through  the  use  of  a  Nearly  Orthogonal  Latin  Hypercube  (NOLH) 
design.  A  NOLH  design  accomplishes  this  by  taking  a  fully  filled  sample  space 
and  strategically  selecting  design  points  to  run  computations  on. 

Figure  7  depicts  this  methodical  approach  pioneered  in  NOLH  designs. 
Specifically,  while  only  a  handful  of  specific  design  points  were  selected  from  the 
original  ll2  factorial  design,  the  output  measures  provided  by  the  response 
surfaces  graph  indicate  the  general  areas  of  peak  performance.  Additionally,  its 
11  computations  are  a  full  magnitude  lower  than  the  121  computations  required 
in  a  ll2  factorial  design. 
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Figure  7.  NOLH  Example.  Source:  Sanchez  &  Wan  (2012). 


Understanding  where  design  points  are  being  placed  within  a 
experimental  space  is  an  important  part  of  any  DOE  design,  especially  a  NOLH. 
One  method  to  show  this  layout  is  through  the  use  of  a  scatterplot  matrix 
(Sanchez  &  Wan,  2012).  A  scatterplot  matrix  is  similar  to  the  two-dimensional 
graphs  shown  in  Figure  7,  but  expands  off  this  to  include  numerous  additional 
dimensions.  Figure  8  shows  Factorial  and  NOLH  designs  and  their  respective 
abilities  to  fill  the  design  space. 


17 


Figure  8.  Space  Filling  Designs.  Source:  Sanchez  &  Wan  (2012). 

In  this  thesis,  DOE  is  used  to  gain  insight  on  the  effects  of  MSM’s  inputs 
on  its  various  output  measurements,  the  response  surfaces.  This  experiment 
used  24  factors  which  were  varied  at  chosen  levels  to  produce  a  257  design 
point  NOLH  DOE  design.  A  NOLH  design  was  chosen  due  to  the  quantity  of 
inputs  that  drive  the  MSM,  and  in  real  life  the  HRDP.  A  more  detailed  description 
of  DOEs  usage  for  evaluation  and  analysis  is  revealed  in  Chapter  IV.  In  closing, 
using  DOE  as  an  evaluation  tool  allows  a  user  to  identify  a  model’s  inputs  cause- 
effect  relationship  with  the  model’s  outputs,  and  therefore  facilitates  the  ability  to 
derive  how  the  processes  in  a  model  are  interacting  at  an  abstract  level. 
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III.  MSM  SIMULATION  MODEL 


The  purpose  of  this  chapter  is  to  annotate  the  architecture  and 
mechanisms  that  make  up  the  MSM  model.  We  first  examine  the  MSM  from  a 
high-level  approach,  starting  with  the  methodology  used  in  constructing  the  MSM 
and  giving  an  overview  of  how  the  program  runs.  We  then  move  into  examining 
the  finer  mechanisms  that  drive  the  MSM.  This  includes  the  various  modeling 
controls  that  facilitate  the  execution  of  the  program,  the  inputs  that  set 
parameters  and  establish  data  table  look-ups,  as  well  as  dissecting  MSM’s 
simulation  processes  to  show  the  individual  components  that  drive  the  model. 
Finally,  we  examine  the  outputs  the  MSM  produces. 

A.  DEVELOPMENT  APPROACH 

The  MSM  is  a  computer  simulation  model,  programmed  in  the  Java 
language  that  uses  the  capabilities  associated  with  agent-based  simulation  to 
capture  various  manpower  metrics  during  its  simulation.  For  the  MSM,  Marine 
Agents  serve  as  the  individual  agent  instances  that  navigation  through  the 
simulation.  A  bottom  up  development  methodology  was  used  when  constructing 
the  MSM.  With  this  approach,  the  developer  identified  the  numerous  individual 
manpower  processes  of  the  HRDP,  built  them  as  Java  classes  with  their  own 
state  properties  and  unique  behavioral  methods,  and  then  aggregated  these 
processes  to  establish  the  manpower  throughput  of  the  FIRDP. 

B.  PROGRAM  EXECUTION  OVERVIEW 

The  processes  that  the  MSM  uses  are  aggregated  in  a  specific  order  to 

accomplish  key  phases  of  the  MSM  program  as  it  runs.  At  a  high-level  view,  this 

aggregation  is  organized  into  five  phases.  These  phases  are  annotated  as  the 

user  input,  database  pull,  system  initialization,  simulation,  and  data  output 

processing.  The  progression  of  these  phases,  with  the  specific  processes  internal 

to  them,  is  shown  in  Figure  9  as  a  flowchart  where  the  rectangles  represent 

phases,  and  the  circle  nodes  inside  them  represent  specific  event  processes.  Out 

19 


of  these  phases,  the  Simulation  Phase  is  the  main  phase  that  drives  the 
simulation.  This  phase  is  responsible  for  simulating  the  careers  of  all  the  Marine 
Agents,  which  ultimately  gets  captured  as  MSM’s  manpower  output 
measurements. 


Figure  9.  MSM  Program  Flowchart.  Adapted  from  Garrick  (2014). 


The  MSM  is  a  time-step  simulation  where  the  metric  for  advancement  is  a 
one-year  increment  within  the  simulation  phase.  This  means  that  all  Marine 
Agents  complete  a  singular  process  before  being  shuffled  to  the  next  process.  As 
such,  each  Marine  Agent  is  subject  to  losses,  gains,  promotions,  lateral  moves, 
and  accessions  once  per  fiscal  year  before  the  simulation  increments  by  a  fiscal 
year  and  repeats  itself.  This  process  progression  can  therefore  be  viewed  as  a 
waterfall  structure;  once  one  process  is  completed  all  Marine  Agents  move 
forward  as  a  group  to  the  next  process.  The  inner  components  of  these 
processes  are  documented  as  event  graphs  in  Section  E  of  this  chapter. 
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C.  MODELING  CONTROLS 

Facilitating  the  execution  of  the  MSM  are  the  simulation  controls  that 
guide  Marine  Agents  through  their  simulated  careers.  These  controls  include  the 
network  that  Marine  Agents  use  to  advance  in  their  careers,  the  rules  and 
conditions  that  bound  the  Marine  Agent  to  actual  HRDP  processes,  and  the 
usage  of  randomness  to  simulate  non-deterministic  decisions  that  occur  within 
the  HRDP. 

1.  Structure 

The  foundational  control  mechanism  of  the  MSM  is  the  state  graph 
network.  This  network  represents  the  actual  manpower  structure  of  the  Marine 
Corps  where  each  state  is  associated  to  a  direct  MOS-Grade  pair.  This 
architecture  was  specifically  chosen  as  it  mimics  the  GAR.  As  previously 
discussed  in  Chapter  I,  the  GAR  is  a  manpower  document  that  sets  target 
quantities  for  every  MOS-Grade  pair  in  order  to  account  for  the  Marine  Corps 
range  of  manpower  specialties  and  grade-experience  while  conforming  to  that 
fiscal  year’s  end  strength  ceiling  constraint.  A  snapshot  of  a  GAR  can  be  viewed 
in  Appendix  B.  These  MOS-Grade  states  are  connected  by  a  directed  graph. 
This  directed  graph  specifies  which  MOS-Grade  state  a  Marine  Agent  can 
transition  to  as  it  simulates  its  career.  Figure  10  illustrates  a  snapshot  of  this 
MOS-Grade  state  directed  graph.  Specifically,  states  are  shown  as  circles  and 
the  specific  MOS-Grade  a  particular  state  is  representing  is  annotated  in  its 
center.  Arrows  link  states  to  each  other,  with  the  arrows  direction  representing 
the  transition  direction  a  Marine  Agent  can  make.  When  aggregated  with  all  MOS 
and  Grade  combinations,  these  states  and  transitions  make  up  the  directed 
graph  network  for  the  MSM.  With  respect  to  constructing  this  network,  the  MSM 
uses  a  GAR  file  input  to  instantiate  all  the  MOS-Grade  states  for  a  simulation, 
and  a  MOS  data  Excel  spreadsheet  input  to  establish  the  state  to  state  transition 
possibilities.  More  information  regarding  these  inputs  is  revealed  in  Section  D  of 
this  chapter. 
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Figure  10.  MOS-Grade  State  Network  Snapshot. 

Adapted  from  Garrick  (2014). 


Figure  10  also  shows  the  progression  for  MOSs  from  a  recruit  to  a  primary 
MOS.  The  four  categories  of  MOS  progression  are:  Recruit  MOS,  Basic  MOS, 
Entry  Level  MOS,  and  Primary  MOS.  An  important  item  to  note  is  that  the  GAR 
input  does  not  specify  data  for  El  and  E2  grades,  which  means  the  MSM  does 
not  instantiate  those  Recruit  MOS  states.  To  circumvent  this  data  interface  the 
MSM  translates  the  distribution  of  Basic  MOSs  into  probabilities  for  the  E3  grade 
sample  space,  and  then  randomly  selects  a  MOS  to  assign  a  newly  instantiated 
Marine  Agent  to.  This  meets  the  intent  of  the  GAR  as  greater  demand  MOSs 
have  a  higher  likelihood  of  being  selected  by  the  MSM’s  random  number 
generator.  However,  this  initialization  for  El  and  E2  grades  only  occurs  once 
when  a  simulation  run  is  invoked.  The  remainder  of  MSM’s  program  execution 
uses  the  accessions  process  to  properly  simulate  for  these  entry  grades.  More 
information  about  the  MSM’s  processes  is  discussed  in  Section  E  of  this  chapter. 
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Besides  for  providing  the  progression  network  that  Marine  Agents 
navigate  through,  all  MOS-Grade  states  retain  specific  functionality  within  the 
MSM.  First,  they  act  like  a  container  bin.  All  MOS-Grade  states  keep  a  tally  of 
Marine  Agents  that  occupy  that  state.  Conditional  rules  within  the  MSM’s  source 
code  require  space  availability  within  a  MOS-Grade  state  for  a  Marine  Agent  to 
transition  into  it.  This  ceiling  is  established  within  the  MSM  by  the  GAR  input. 
Additionally,  MOS-Grade  states  retain  metadata  for  that  state.  This  data 
eventually  becomes  the  predominance  of  the  output  measurements  produced  by 
the  MSM.  Metadata  that  each  MOS-Grade  state  retains  includes: 

•  Predecessor  states  that  are  connected  to  it. 

•  Successive  states  that  a  particular  state  can  direct  to. 

•  Loss  and  Gain  type  categories. 

•  Quantity  of  losses  and  gains,  per  HRDP  subcategories,  captured  in 
monthly  increments. 

•  Retention  metrics;  to  include  quantity  of  Marine  Agents  eligible  for 
retention  and  quantity  of  Marine  Agents  actually  retained. 

•  Promotion  metrics;  to  include  quantity  of  Marine  Agents  eligible  for 
promotion  and  quantity  of  Marine  Agents  actually  promoted. 

2.  Deterministic  Processing 

Reinforcing  the  foundational  control  provided  by  the  MOS-Grade  state 
directed  graph  network  are  the  deterministic  processes  that  drive  Marine  Agents 
through  the  simulation.  These  processes  are  executed  through  rule-based 
decisions  and  are  utilized  when  the  outcome  of  a  decision  can  be  simplified  into 
a  binary  option.  Boolean  conditions  are  leveraged  within  if-else  statements, 
switch  statements,  for-loops  or  while-loops,  and  many  other  methods  to  navigate 
a  particular  process’s  algorithm.  This  control  approach  is  deterministic  because 
an  identical  Marine  Agent,  under  the  same  inputs  and  same  parameters,  will 
select  the  exact  same  edge  transitions  to  navigate  down  the  same  career  path 
every  time.  A  few  examples  of  binary  rules  the  MSM  uses  to  select  edge 
transitions  are: 
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•  Has  the  Marine  Agent  received  the  maximum  allotment  of 
promotion  Passovers? 

•  Can  the  Marine  Agent  lateral  move  to  a  new  MOS? 

•  Is  there  availability  within  a  particular  MOS-grade  state  for  a  Marine 
Agent  to  transition  into? 

3.  Non-deterministic  Processing 

Non-deterministic  processing  can  be  categorized  as  the  process  where 
given  the  exact  same  input  conditions  when  repeating  a  process,  a  different 
output  verdict  can  be  selected  (Sanchez,  2007).  This  is  also  the  process  that 
gives  the  MSM  its  stochastic  modeling  functionality.  Within  the  MSM,  this  is  seen 
in  situations  where  identical  Marine  Agents,  under  the  exact  same  conditions  and 
inputs,  will  select  different  edge  transitions  to  navigate  down  for  their  simulated 
career.  In  event  graph  terminology,  this  is  the  process  of  selecting  a  transition 
edge  that  will  identify  the  specific  next  event  that  will  be  executed. 

Non-deterministic  processing  is  used  within  the  MSM  to  simulate 
decisions  a  Marine  Agent  would  have  to  make;  an  example  of  such  would  be 
deciding  to  reenlist  or  transition  out  of  active  service  at  the  expiration  of  their 
enlistment  contract.  Non-deterministic  processing  is  also  leveraged  to  select 
Marines  at  various  boards  within  the  HRDP;  an  example  would  be  the  final 
selection  process  of  selecting  Marines  to  be  promoted  from  a  sampling  pool  filled 
with  qualified  candidates.  Non-deterministic  processing  is  also  used  to 
incorporate  non-idealistic  random  events  that  occur  within  the  Marine  Corps 
HRDP  process;  cases  of  deaths,  deserters,  or  legal  holds  just  to  name  a  few. 

The  execution  of  non-deterministic  processing  is  carried  out  by  employing 
the  Mersenne  Twister  random  number  generator  in  two  different  methods.  We 
will  classify  these  two  different  methods  as  a  percentage  line  selector  and  a 
Bernoulli  Trial.  The  percentage  line  selector  incorporates  most  the  characteristics 
of  a  Markov  Chain  model,  however  for  only  one  transition  instance  per 
invocation. 
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In  a  percentage  line  selector,  the  potential  outcomes  are  categorized  and 
bounded  on  a  percentage  line  that  ranges  from  0  to  100  percent.  The  outcomes 
are  weighted  such  that  events  that  are  more  likely  to  occur  will  receive  a  larger 
range  selection  on  the  percentage  line.  This  weighted  value  is  determined  by 
historical  data  that  is  input  into  the  MSM  and  translated  into  a  percentage  rate 
value.  All  possible  transition  events  are  then  positioned  on  this  percentage  line 
such  that  they  are  all  mutually  exclusive,  they  are  fully  married  next  to  one 
another,  and  they  are  normalized  to  fit  the  entirety  of  the  percentage  line.  Finally, 
the  random  number  generator  generates  a  random  percentage  that  will  map  to  a 
singular  outcome.  This  outcome  is  the  decision  point  for  that  event  and  maps  to 
the  edge  transition  that  particular  Marine  Agent  will  navigate  down.  Figure  1 1 
illustrates  this  method. 
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Loss  subcategories  are  not  drawn  to  scale,  and  their  respective  values  were 
arbitrary  created  for  this  illustration.  Intent  of  this  illustration  is  to  show  how  the 
loss  subcategories  marry  up  on  a  number  line,  that  they  sum  to  100%,  and  that 
the  Mersenne  generates  a  random  “pLoss”  percentage  used  to  execute  one  of 
the  potential  subcategories. 

Figure  1 1 .  Loss  Subcategories  Viewed  on  Number  Line. 

Adapted  from  Garrick  (2014). 


The  second  non-deterministic  processing  method  used  is  a  Bernoulli  trial. 
This  method  is  used  specifically  within  the  MSM  to  simulate  the  occurrence  of 
those  non-idealistic  events,  as  well  as  other  unique  success  and  failure  cases.  In 
this  method,  a  look-up  table  is  referenced  to  set  a  threshold  that  will  serve  to 
delineate  a  trail’s  success  or  failure.  This  success  or  failure  is  then  mapped  to  the 
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event  decision.  The  Merseene  Twister  is  again  used  to  generate  a  random 
number  that  is  translated  to  a  percentage,  and  then  referenced  to  the  Bernoulli 
trial  to  select  a  course  of  action  for  that  singular  trail. 

D.  MSM’S  INPUTS 

The  MSM  requires  numerous  inputs  in  order  to  run  its  program.  The 
insertion  of  these  inputs  occurs  at  two  file/folder  locations,  where  the  data 
inserted  at  these  locations  are  either  directly  translated  to  parameters  within  the 
simulation  or  are  used  to  reference  Marine  Corps  online  manpower  archive 
databases.  Figure  12  illustrates  these  inputs  in  a  tree  structure.  The  nodes 
outlined  in  red  illustrate  which  inputs  are  directly  translated  into  MSM 
parameters.  The  nodes  outlined  in  blue  illustrate  which  inputs  are  used  to 
reference  online  databases.  The  bullets  within  the  tree  show  the  itemized  inputs. 


Figure  12.  MSM  Inputs  Shown  in  Tree  Structure. 

Adapted  from  Garrick  (2014). 
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1.  Direct  Inputs 

The  first  location  that  direct  inputs  are  inserted  into  is  the 
“\\MSM\input\gar_data”  folder.  Within  this  folder,  the  user  will  manually  insert 
GAR  files  into  the  folder  where  each  file  yields  information  for  a  single  fiscal  year. 
These  file  names  must  be  changed  to  follow  a  standard  naming  convention  of 
“nnenl<FY>.gbl”  and  “nnenl<FY>.tre”  for  the  GAR  build  reports  and  allocation  of 
training  reports,  respectively,  where  “<FY>”  is  replaced  with  the  two-digit  fiscal 
year  the  document  represents.  There  is  no  limit  to  the  number  of  GAR  files 
inserted  into  this  folder,  only  that  GAR  file  fiscal  years  need  to  match  the  range  of 
years  the  MSM  program  will  forecast  through.  However,  if  the  user  chooses  to 
forecast  into  the  future  where  no  GAR  file  exists,  the  MSM  will  reuse  its  previous 
FY  GAR  input  for  the  next  simulated  year.  It  is  critical  that  these  preparatory 
details  are  followed  as  the  MSM’s  source  code  is  hard-coded  to  look  exactly  in 
that  folder  for  exactly  those  file  names.  If  the  file  names  are  in  the  wrong  location 
or  labeled  incorrectly,  the  MSM  program  will  not  be  able  to  pull  the  required  data 
and  construct  its  MOS-Grade  state  network,  which  will  therefore  crash  the 
program. 

The  other  direct  inputs  are  all  inserted  into  the  “siminputs”  Excel  file.  This 
document  is  broken  down  into  eight  sheets,  where  each  sheet  is  associated  with 
a  particular  HRDP  process  its  inputs  will  influence.  The  following  sheets  require 
values  to  be  input  by  the  user: 

•  Main:  includes  simulation  forecast  y-years  for  k-repetitions 

•  Career  force  controls 

•  Pay  data 

•  Monthly  boot  camp  shipment  distributions 

•  MOS  data.  This  sheet  is  responsible  for  providing  the  series  of 
rules  that  affect  Marine  Agents  and  how  they  transition  to  the 
various  MOS-Grade  states 

The  Excel  file  “siminputs”  also  contains  sheets  where  the  user  is  afforded 

the  option  to  insert  non-mandatory  inputs.  These  inputs  allow  the  user  to  quickly 
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alter  some  of  the  simulation  parameters  to  specifically  target  certain  case 
scenarios.  Section  E  of  this  chapter  will  go  into  much  further  detail,  but  at  an 
overview  level,  some  of  the  MSM’s  Simulation  Phase  processes  use  algorithms 
that  invoke  the  functionally  to  scale  the  results  for  a  particular  process.  What  this 
means  is  if  the  optional  user  data  input  is  higher  or  lower  than  what  the  MSM  is 
using  as  its  baseline  value,  the  MSM  will  increase  or  decrease  its  results, 
respectively.  This  scaling  is  also  done  proportionally  to  the  percentage  difference 
between  the  user  input  and  the  MSM’s  baseline  value.  A  perfect  example  of  this 
functionality  is  if  a  user  wanted  to  re-run  a  simulation  but  for  a  different  targeted 
fiscal  year  end  strength.  The  user  could  quickly  insert  that  value  into  the  optional 
spreadsheet  and  receive  an  ad  hoc  assessment  without  having  to  go  through  the 
lengthy  process  of  completely  rebuilding  the  GAR  files.  The  sheets  these  optional 
inputs  can  be  found  in  are: 

•  Fiscal  Year  Targets 

•  MOS  Targets 

•  MOS  and  Grade  Targets 

Additional  details  regarding  these  inputs  can  be  found  in  Appendix  A. 

2.  Reference  Inputs 

The  other  type  of  input  that  facilitates  the  MSM’s  functions  is  the  reference 
input.  These  indirect  inputs  are  the  historical  data  the  MSM  pulls  from  an  online 
Oracle  database.  Two  types  of  data  base  pulls  occur  here:  acquiring  historical 
data  of  manpower  pyramid  transition  rates,  and  inventory.  These  database  pulls 
provide  historical  data  that  the  MSM  uses  to  compute  its  various  probabilities  of  a 
Marine  Agent  executing  a  specific  edge  transition.  The  MSM  facilitates  the  ability 
to  pull  from  two  different  databases:  Total  Force  Data  Warehouse  (TFDW)  and 
Operational  Data  Store  Enterprise  (ODSE).  TFDW  and  ODSE  are  very  similar  in 
the  data  they  contain.  The  TFDW  stores  historical  information  at  the  one-month 
increment.  Specific  month’s  data  can  be  obtained  by  providing  the  specific 
sequence  number  mapped  to  that  month.  Conversely,  the  ODSE  maintains  real- 
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time  manpower  metrics  vice  historical  data,  therefore  providing  the  user  an  exact 
inventory  of  today’s  force.  The  data  pulled  from  these  databases  includes: 

•  Loss  data 

•  Reenlistment  data 

•  Promotion  data 

•  Gains  data 

•  Recruit  loss  data 

The  other  type  of  database  pull  is  the  inventory  pull.  This  pull  obtains  a 
real  sight  picture  of  the  manpower  layout  within  the  Marine  Corps  at  a  specific 
point  in  time.  This  means  each  real  historical  Marine  becomes  represented  as  a 
Marine  Agent  in  order  to  initiate  the  MSM.  The  attributes  for  each  Marine  pulled 
are: 

•  Primary  military  occupational  specialty  (PMOS) 

•  Gender 

•  Years  of  service  (YOS) 

•  Time  in  grade  (TIG) 

•  Expiration  of  current  contract  (ECC)  Year 

•  ECC  Month 

•  Grade  Selected  For  (if  pending  an  approved  promotion) 

E.  SIMULATION  PHASE 

As  Figure  9  showed,  there  are  five  phases  that  make  up  the  MSM 
program.  The  first  three  phases  are  responsible  for  establishing  the  MSM’s 
environment.  This  includes  reading  in  user  inputs,  pulling  reference  data  from 
online  data  repositories  and  storing  that  information  within  MSM  local  data  tables, 
and  then  initializing  the  MSM  MOS-Grade  state  graph  network  in  terms  of 
constructing  the  network  and  then  populating  it  with  the  initial  distribution  of 
Marine  Agents.  Once  these  three  phases  conclude,  the  program  is  ready  to  enter 
the  simulation  phase. 
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The  simulation  phase  is  the  main  phase  of  the  MSM  as  it  is  responsible  for 
simulating  the  specific  HRDP  processes.  This  phase  simulates  for  a  user 
specified  number  of  years.  Specifically,  after  the  last  process  within  the  phase 
has  concluded,  the  MSM  either  will  increment  to  the  next  fiscal  year  and  repeat 
the  simulation  phase  or,  if  all  forecast  years  have  been  simulated,  the  MSM  will 
increment  to  the  data  aggregation  phase.  When  simulating  a  single-year 
iteration,  the  following  manpower  processes  are  executed  in  the  following 
sequence: 

1 .  Losses/Retentions 

2.  Gains 

3.  Lateral  move  with  simultaneous  promotions 

4.  Lateral  moves 

5.  Promotions 

6.  Accessions 

After  the  sixth  and  final  process,  the  program  stores  its  calculated  metrics 
and  determines  its  next  course  of  action.  If  there  are  still  additional  years  left  to 
forecast,  the  program  iterates  and  continues  simulating  for  the  following  year.  If 
the  MSM  has  concluded  its  final  year  forecast,  the  simulation  resets  all  its  values 
and  will  repeat  for  a  user  specified  number  of  times.  This  repetition  allows  for  the 
stochastic  properties  of  the  simulation  to  take  effect. 

1.  Losses  Process 

This  process  of  the  simulation  model  handles  the  attrition  aspect  of  the 
HRDP.  Each  Marine  Agent  within  the  simulation  is  pulled  and  run  through  a  trial 
to  determine  whether  that  Marine  Agent  continues  their  simulated  career  or  if 
they  have  been  randomly  selected  to  exit  the  Marine  Corps.  In  the  losses 
process  event  graph,  there  are  two  major  decisive  event-nodes  that  occur.  The 
first  examines  the  Marine  Agents  years  of  service  (YOS)  in  a  simple  if-else 
statement.  If  the  Marine  Agent’s  YOS  exceeds  the  maximum  time  in  service 

(maxTIS)  constraint,  as  annotated  by  the  user  in  their  ECFC  inputs,  then  the 
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Marine  Agent  has  reached  service  limitations  and  is  force  to  exit  the  Marine 
Corps.  If  cleared,  the  Marine  Agent  continues  to  the  second  decision  node  where 
they  undergo  a  percentage  line  selector,  non-deterministic  trial,  to  determine  if 
they  will  be  categorized  as  one  of  the  possible  losses,  or  if  they  will  continue  their 
simulated  careers.  This  trial  references  MSM’s  data  tables,  which  were 
constructed  by  the  reference  input-type,  to  look  up  historical  loss  percentages  for 
the  various  loss  categories.  Data  for  each  loss  category  is  categorized  by  three 
parameters.  The  three  parameters  used  are: 

•  Grade 

•  Inventory  type:  if  the  Marine  Agent’s  ECC  is  the  current  simulation 
year,  they  are  annotated  as  “eas-0.”  If  the  ECC  is  the  following 
simulation  year,  they  are  annotated  as  “eas-1.”  And  if  their  ECC  is 
beyond  that,  they  are  annotated  as  “neas.” 

•  Retirement  condition:  if  the  Marine  Agent  is  eligible  for  retirement, 
they  are  annotated  as  “re.”  Otherwise,  they  are  annotated  as  “nre.” 

The  percentage  is  calculated  by  dividing  a  loss  category  by  the  total 
inventory.  Whichever  edge  condition  meets  the  generated  percentage  becomes 
the  edge  transition  that  is  executed.  Reference  Figure  1 1  to  see  an  illustration  of 
the  percentage  line  selector  non-deterministic  method.  If  the  Marine  Agent 
transitions  to  a  lost  event,  they  are  removed  from  the  MOS-Grade  state  graph 
network,  which  creates  inventory  vacancies  for  that  specific  MOS-Grade.  This 
process  is  shown  in  Figure  12  as  an  event  graph. 
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bootP  =  getBootProb(  grade,  EAS,  ret_EI  ) 
retP  =  getRetireProb(  grade,  EAS,  ret_EI ) 
neasP  =  getNeasProb(  grade,  EAS,  ret_EI ) 
deathP  =  getDeathProb(  grade,  EAS,  ret_EI ) 
e2oP  =  getE2oProb(  grade,  EAS,  ret_EI ) 
otherP  =  getOtherProb(  grade,  EAS,  ret_EI  ) 
easP  =  getEasProb(  grade,  EAS,  ret_EI  ) 
reupP  =  getReupProb(  grade,  EAS,  ret_EI ) 
contP  =  getContProb(  grade,  EAS,  ret_EI ) 
rP  =  getRandomProbQ 


Figure  1 3.  Losses  Event  Graph.  Adapted  from  Garrick  (201 4). 


Before  this  process  ends,  the  model  checks  to  see  if  the  optional  input 
retention  quantity  was  entered  by  the  user.  If  so,  the  program  will  check  to  see  if 
its  original  calculation  has  exceeded  this  user  input  threshold.  If  a  zone  is  under 
its  user  defined  retention  zone  threshold,  no  further  action  will  occur.  If  a  zone  is 
over  its  threshold,  the  program  will  randomly  select  a  Marine  Agent  from  the 
retention  container,  re-classify  them  as  EAS,  and  remove  them  from  the  state- 
graph.  This  itemized  removal  continues  until  the  specified  retention  zone  is  under 
its  maximum  threshold. 


2.  Gains  Process 

The  Gains  Process  simulates  the  real  world  scenarios  when  prior  service 
Marines  return  back  into  the  Marine  Corps  inventory  and  are  counted  toward  the 
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Marine  Corps’  overall  end  strength.  As  described  in  Chapter  I,  this  can  occur  for 
a  variety  of  reasons;  such  as  returning  from  recruiting  duty  or  returning  from  a 
medical  hold.  This  process  is  executed  by  a  Bernoulli,  non-deterministic,  trial. 
Specifically,  every  time  a  Marine  Agent  is  tried  in  the  losses  process,  the  gains 
process  is  invoked  and  ran  in  parallel.  One  Bernoulli  trial  is  run  for  each  possible 
gain  category  (gainCat)  per  each  process  invoked.  Look  up  tables,  which  were 
constructed  by  the  MSM’s  reference  inputs,  yield  the  trial  success  probabilities 
for  each  gainCat.  Three  parameters  are  used  to  determine  the  specific 
percentage  to  use  for  each  trial: 

•  gainCat 

•  Grade 

•  Base  inventory  for  that  grade 

The  percentage  is  calculated  by  dividing  the  quantity  of  a  particular 
gainCat  for  the  identified  grade  by  the  base  inventory  for  that  grade.  Figure  14 
displays  the  event  graph  for  the  Gains  Process. 
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Figure  1 4.  Gains  Event  Graph.  Adapted  from  Garrick  (201 4). 

If  the  Bernoulli  trial  was  successful,  the  MSM  will  instantiate  a  new  Marine 
Agent  and  insert  them  into  the  MOS-grade  state  network.  To  accomplish  this,  the 
MSM  randomly  generates  all  required  Marine  Agent  attributes  which  include: 

•  YOS 

•  Time  in  grade  (TIG) 

•  ECC 

•  PMOS 

•  Grade 

•  Month  gained 

3.  Lateral  Move  Promotions  Process 

The  Lateral  Move  Promotions  Process  is  the  third  process  invoked  during 
the  Simulation  Phase.  It  combines  the  lateral  move  process,  where  a  Marine 
Agent  obtains  a  new  MOS  within  the  same  grade,  with  the  promotions  process, 
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where  a  Marine  Agent  obtains  a  higher  grade  but  within  the  same  MOS.  This  can 
be  summarized  as  a  simultaneous  horizontal  and  vertical  move  within  the  MOS- 
Grade  state  graph  network.  This  process  is  unique  in  that  it  only  targets  the  small 
population  of  Marine  Agents  with  MOSs  that  allow  for  career  progression  to  new 
MOSs,  lateral  moves.  This  rule  constraint  is  constructed  from  the  direct  input 
“FY<FY>  MOS  data,”  which  is  an  Excel  sheet  that  resides  within  the  “siminputs” 
Excel  document. 

The  lateral  move  promotions  process  was  strategically  placed  to  precede 
the  regular  lateral  move  and  regular  promotion  processes  due  to  the  inventory 
quantity  of  Marine  Agents  they  each  contain.  This  means  Marine  Agents  that 
have  the  potential  for  a  new  MOS  are  given  the  opportunity  to  fill  vacant  billet 
spots  in  other  MOS-Grade  states  before  Marine  Agents  with  standard  career 
progressions.  If  these  processes  were  reversed  there  would  be  a  much  greater 
probability  that  all  potential  MOS-Grade  states  would  be  filled  by  standard  career 
progression  Marine  Agents,  leaving  lateral  move  promotable  Marine  Agent 
candidates  stagnant. 

The  Lateral  Move  Promotions  Process  is  also  the  first  process  where  the 
procedural  priority  for  processing  Marine  Agents  is  critical.  This  stems  from  the 
constraint  that  a  state  must  possess  billet  vacancies  in  it  to  allow  a  Marine  Agent 
to  transition  to  it.  To  accommodate  this,  the  MSM  examines  Marine  Agents  in 
reverse  grade  order;  starting  with  the  grade  E9  and  then  working  downwards. 
This  allows  Marine  Agents  to  move  into  higher  billet  vacancies  immediately, 
which  therefore  creates  a  billet  vacancy  in  the  MOS-Grade  state  they  just  left. 
This  effect  trickles  downward  to  the  lowest  grade  of  Marine  Agents. 

The  algorithm  for  the  lateral  move  promotions  process  contains  five  steps. 
Figure  15  shows  this  process  in  event  graph  form.  In  procedural  order,  the 
algorithm’s  steps  are  annotated  as: 

1 .  Select  a  MOS-Grade  state  for  Marine  Agents  to  lateral  move  and 
promote  into. 
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2.  Build  a  list  of  Marine  Agents  that  are  allowed  to  be  laterally  moved 
and  promoted  into  the  identified  MOS-Grade  state. 

3.  Count  the  number  billet  vacancies  within  the  targeted  MOS-Grade 
state.  This  is  accomplished  by  obtaining  the  delta  between  current 
occupancy  and  maximum  allowed  occupancy  as  dictated  by  the 
GAR  input. 

4.  Check  if  the  optional  input  “End  Strength”  was  entered  by  the  user. 
If  it  was,  the  MSM  will  take  the  percentage  difference  between  the 
GAR’s  total  strength  and  the  user’s  optional  input  End  Strength, 
and  apply  that  percentage  to  the  MOS-Grade  states  to  increase  or 
decrease  the  total  amount  of  billet  vacancies  respectively.  If  the 
user  did  not  enact  this  option,  the  program  will  use  the  number  of 
billet  vacancies  found  in  step  3. 

5.  While  billet  vacancies  and  Marine  Agents  exist,  randomly  select  a 
Marine  Agent  to  execute  the  lateral  move  and  promotion  transition. 


(next  toGrade  exists) 


(next  toMos  exists) 


(next  toMos  exists) 


eligibleList  =buildl_ist(  toMos,  toGrade)  marine  =  eligibleList.random()  nodeVac  =  nodeVac  - 1 

nodeVac  =  garVacancies  (toMos,  toGrade) 


Figure  15.  Lateral  Move  Promotion  Event  Graph. 

Adapted  from  Garrick  (2014). 


4.  Lateral  Move  Process 

The  lateral  move  process  follows  the  exact  same  methodology  as  the 
lateral  move  promotions  process  except  for  two  critical  areas.  First,  the  process 
will  only  move  Marine  Agents  horizontally  to  a  new  MOS;  so  the  Marine  Agent 
will  maintain  their  same  grade.  Secondly,  this  process  incorporates  P2T2 
possibilities  into  its  algorithm.  Specifically,  being  selected  by  the  board  does  not 
guarantee  the  Marine  Agent  will  transition  to  their  updated  MOS-Grade  state. 


36 


Instead,  the  algorithm  simulates  whether  or  not  the  Marine  Agent  will  be  placed 
back  in  the  state-graph  or  if  they  will  be  placed  into  the  P2T2  container  for  that 
fiscal  year.  This  decision  is  made  by  referencing  the  length  of  the  school,  time  to 
train  MOS  metric,  the  Marine  Agent  would  have  to  attend  to  learn  their  new  trade. 
If  the  school  is  longer  than  365  days,  one-year,  the  algorithm  directly  places  the 
Marine  Agent  into  the  P2T2  container  since  they  will  be  without  a  doubt  still  in 
school  when  that  fiscal  year  ends.  If  the  school  length  is  shorter,  the  algorithm 
calculates  the  probability  of  being  at  that  school  at  any  particular  point  in  the 
fiscal  year  where  the  probably  is  the  duration  of  the  school  divided  by  365  days. 
Once  calculated,  this  percentage  is  passed  to  a  Bernoulli,  non-deterministic,  trial 
where  one  trial  is  run.  If  the  Bernoulli  trial  was  successful,  then  the  model  will 
place  the  Marine  Agent  into  the  P2T2  container.  Otherwise,  they  will  return  to  the 
state-graph  at  their  updated  MOS-Grade  state.  Figure  16  shows  this  process  as 
an  event  graph. 


Figure  1 6.  LatMove  Event  Graph.  Adapted  from  Garrick  (201 4). 

5.  Promotions  Process 

This  process  compliments  the  previously  stated  lateral  move  promotions 

process  in  that  it  is  the  mutually  exclusive  counterpart  to  the  lateral  mover 

indicator.  Specifically,  this  process  focuses  promoting  Marine  Agents  who  do  not 
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possess  MOSs  that  allow  for  lateral  moves.  Non-lateral  move  MOSs  comprise  of 
approximately  95%  of  the  total  MOS  inventory  (Spafford,  2016).  Therefore,  it  is  to 
no  surprise  that  the  algorithm  steps  and  event  graph,  shown  in  Figure  17,  are 
identical. 


(next  toGrade  exists) 


(next  toMos  exists) 


(next  toMos  exists) 


Figure  17.  Promotions  Event  Graph.  Adapted  from  Garrick  (2014). 

6.  Accessions  Process 

The  accessions  process  is  the  final  process  of  the  simulation  phase  and  is 
predominately  summarized  as  the  overall  backfill  of  all  MOS-Grade  state 
vacancies  that  have  trickled  their  way  down  to  the  junior  grade  levels.  The 
algorithm  is  conducted  by  executing  the  following  steps. 

1.  Set  baseline  accession  targets  per  MOS:  This  is  accomplished  by 
aggregating  the  current  inventory  in  the  MOS-Grade  state  graph 
network  for  grades  El  through  E3,  categorizing  by  MOS.  It  then 
subtracts  that  MOS’s  entry  level  aggregate  from  the  GAR’s  allowed 
quantity,  which  is  the  GAR  column  labeled  “E3.”  The  initial 
aggregation  of  the  MSM’s  current  entry  level  inventory  is  required 
due  to  the  data  interface  constraint  with  the  GAR;  that  is  the  GAR 
does  not  identify  MOS  quantities  for  grades  El  and  E2. 

2.  Distribute  MOS  targets  to  entry  level  grades:  By  default,  the  MSM 
takes  step  1’s  deliverable  and  distributes  them  to  the  paygrades 
El ,  E2,  E3,  within  that  MOS,  by  the  constant  ratio  75.84%,  24.15%, 
and  0.01%,  respectively.  However,  if  the  user  set  the  optional  MOS 
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accession  targets  in  the  siminputs  Excel  document,  the  paygrade 
distribution  will  be  applied  to  this  quantity  vice  step  1’s  deliverable. 

3.  Set  MOS-Grade  male  and  female  targets:  The  MSM  takes  step  2’s 
deliverable  and  then  divides  that  quantity  into  male  targets  and 
female  targets.  By  default,  this  distribution  is  calculated  by  the 
MSM’s  reference  input  from  the  historical  Marine  Corps  Oracle 
online  databases.  However,  if  the  user  sets  the  optional  female 
accession  targets  in  the  siminputs  Excel  document,  the 
male/female  distribution  will  then  be  proportionately  adjusted  to 
match  the  user’s  target. 

4.  Instantiate  a  Marine  Agent:  For  every  male/female  MOS-Grade 
target,  the  MSM  will  instantiate  a  new  Marine  Agent.  While  the  sex, 
MOS,  and  grade  are  already  determined,  the  MSM  will  use  its 
percentage  line  non-deterministic  selector  to  randomly  determine 
the  contract  length  of  the  Marine  Agent  and  the  Month  in  which  that 
Marine  Agent  entered  boot  camp.  Contract  length  probabilities  are 
set  by  the  required  user  input  ‘FY<FY>  MOS  Data’  sheet  in  the 
Excel  file  “siminputs.” 

5.  Test  for  boot  camp  loss,  and  insert  into  MOS-Grade  state  graph 
network:  Each  instantiated  Marine  Agent  will  be  passed  through  a 
boot  camp  losses  trail.  This  trial  is  very  similar  to  the  Losses 
process  event  graph.  If  a  loss  event  is  not  selected,  then  Marine 
Agent  is  then  inserted  into  their  respective  MOS-Grade  state. 

F.  MODEL  OUTPUTS 

At  the  completion  of  the  Simulation  Phase,  the  program  records  the  data 
associated  with  that  particular  repetition  run.  The  program  will  then  iterate  and 
run  the  entire  simulation  over  again,  using  the  same  inputs,  to  obtain  new  output 
metrics.  This  repetition  allows  for  stochastic  properties  to  take  effect.  Once  all 
repetitions  have  run,  the  program  computes  a  statistical  analysis  of  the  different 
runs.  This  is  called  the  data  aggregation  process.  Each  metric  within  the  process 
is  analyzed  by  its: 

•  Means 

•  Standard  Deviation 
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Maximum 


• 

•  Minimum 

These  output  measures  are  aggregated  at  three  different  tiers,  where 
each  tier  yields  the  same  data  but  at  different  levels  of  fidelity.  Tier  I  data  is  the 
most  abstract  data  output,  where  the  data  obtained  from  all  the  MOS-Grades  are 
aggregated  into  a  fiscal  year  sum.  Tier  II  still  aggregates  its  data  into  fiscal  year 
sums,  however  it  incorporates  three  possible  classifiers  the  data  can  be  sorted 
into:  MOS,  Grade,  or  MOS  and  Grade.  Tier  III  expands  off  of  Tier  II  in  that  it 
keeps  the  same  classifiers,  but  the  data  drills  into  the  monthly  level.  Figure  18 
depicts  this  layered  metric  tier,  and  its  respective  scope  of  detailed  information 
the  Model’s  user  is  wishing  to  obtain.  For  Chapter  Ill’s  Design  of 
Experimentation,  Tier  I  output  data  was  utilized.  For  Chapter  IV’s  Validation,  a 
combination  of  Tier  I  and  Tier  II,  with  categorization  by  grade,  output  data  was 
utilized.  More  information  regarding  the  output  measures,  and  their  respective 
definitions  and  metric  scope,  is  listed  in  Appendix  D. 


USMC  Whole 
Aggregated  by  Year 

Tterl  r— 


Aggregated  by  Year  Aggregated  by  Year  Aggregated  by  Year 
Tier  2  Categorized  by  MOS  Categorized  by  Grade  Categorized  by  MOS  &  Grade 


Aggregated  by  Month  Aggregated  by  Month  Aggregated  by  Month 

Categorized  by  MOS  Categorized  by  Grade  Categorized  by  MOS  &  Grade 


Figure  18.  Depiction  of  MSM’s  Output  Metrics,  Aggregated  at  Different 

Levels.  Adapted  from  Garrick  (2014). 
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IV.  DESIGN  OF  EXPERIMENTS  EVALUATION 


This  chapter  describes  the  design,  procedure,  and  experimental  results 
used  for  conducting  the  design  of  experiments  (DOE)  evaluation.  This  evaluation 
provided  two  key  metrics;  it  verified  the  input-to-output  response  correlation,  and 
it  identified  the  input  significance  priority.  Correlation  is  the  process  of  looking  at 
two  variables  and  determining  if  one  influences  the  other.  This  influence  can  be 
either  strong  or  weak  and  incorporates  the  direction,  positive  or  negative,  toward 
which  the  variables  will  react  (Law  &  Kelton,  2000).  This  strength  and  direction 
relationship  is  often  referred  to  as  the  correlation  gradient  (Law  &  Kelton,  2000). 
Within  this  DOE  evaluation,  gradient  was  the  primary  metric  for  measurement 
used  as  it  allowed  for  cause-effect  HRDP  relationships  to  be  easily  identified  and 
analyzed. 

The  second  evaluation  metric  was  the  input  significance  priority.  Input 
significance  priority  is  the  nomenclature  used  to  identify  which  terms  were 
responsible  for  having  the  most  impact  on  the  produced  metamodel.  This  is 
determined  by  computing  the  ratio  between  the  parameter  coefficient  within  a 
metamodel’s  response  surface  equation  and  its  standard  error  (“Estimates,”  n.d., 
Sorted  Estimates  section).  Terms  with  a  higher  level  of  significance  identify  which 
inputs  are  more  sensitive  in  forecasting  future  data  predictions. 

To  compute  statistics  from  the  DOE  experiment,  we  used  the  statistical 
software  suite  JMP  Pro  12.  An  overview  of  the  functionality  utilized  is  described 
in  Section  B  of  this  Chapter. 

A.  CONSTRAINTS  AND  DESIGN 

This  section  describes  the  constraints  that  were  presented  in  subjecting 
the  MSM  to  a  DOE.  It  highlights  the  decisions  made  to  comply  with  these 
constraints,  to  include  what  the  final  DOE  design  entailed. 
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1. 


Factor  Selection 


DOE  is  naturally  limited  by  the  number  of  factors  it  can  take  in  and  create 
a  design  from.  This  limitation  stems  from  the  computational  costs  required  in 
conducting  an  experiment  with  a  large  set  of  factors  while  simultaneously 
maintaining  the  experiment’s  orthogonality.  The  largest  NOLH  that  exists  today 
allows  for  up  to  29  factors,  which  produces  a  257-design  point  design  (Cioppa  & 
Lucas,  2007).  A  Nearly  Orthogonal  Balanced  design  is  a  different  type  of  DOE 
design  that  does  allow  for  more  factors  to  be  used  (Vieira  Jr.  et  al,  2012). 
However,  this  design  incorporates  a  mix  of  discrete  and  continuous  factors  that 
did  not  interface  well  with  the  MSM  and  therefore  was  discarded. 

Condensing  MSM’s  numerous  inputs  to  map  to  29  DOE  factors  proved 
challenging.  This  problem  stemmed  from  MSM’s  multi-dimensional  look  up  tables 
that  are  constructed  from  its  reference  input  Oracle  database  pulls.  Take  for 
example  the  loss  rate  inputs  for  just  the  loss  subcomponent  EAS.  This 
subcomponent  takes  the  parameters  of  MOS,  Grade,  and  Retirement  Eligibility  to 
look  up  the  specific  EAS  rate  within  those  dimensions.  Just  those  EAS  loss  rates 
yield  4320  inputs;  240  MOS  x  9  Grades  x  2  Retirement  Options  =  4320  inputs. 
Extrapolating  this  multi-dimensionality  across  the  other  HRDP  process  categories 
unquestionably  exceeds  a  NOLH  factor  threshold.  To  circumvent  this  constraint, 
we  implemented  the  use  of  abstraction  within  our  DOE. 

With  the  abstraction  methodology,  we  targeted  the  specific  subcomponent 
of  a  HRDP  process  as  a  single  DOE  factor  vice  assigning  a  factor  to  every  input 
in  its  multi-dimensional  table.  With  this  approach,  the  DOE  factor  acted  as  a 
coefficient  that  adjusted  the  subcomponent  it  was  representing.  Because  this 
subcomponent  was  uniformly  adjusted  every  time  a  value  was  returned  from  the 
look  up  tables,  the  orthogonality  of  the  experiment  was  maintained. 

Table  1  illustrates  the  application  of  this  abstraction  methodology. 
Specifically,  the  table  provides  a  snapshot  of  the  “Continue  Rate”  category,  which 
is  a  subcomponent  of  the  losses  HRDP  process.  As  the  table  shows,  its  multi- 


42 


dimensionality  yields  six  inputs  for  just  a  single  grade  return.  The  original  inputs 
are  annotated  in  the  column  “Continue  Rate”  and  serve  as  the  baseline.  The 
column  “NOLH  Factor”  reflects  what  that  DOE  factor  is  set  at  for  that  particular 
design  point.  Acting  as  a  coefficient,  the  “NOLH  Factor”  is  multiplied  with  the 
baseline  to  produce  the  adjusted  input;  labeled  “Function  Rate.”  This  adjusted 
rate  is  what  is  returned  to  the  function,  allowing  that  HRDP  process’s  algorithm  to 
continue  its  execution  in  alignment  with  the  DOE  design  point. 


Table  1 .  Snapshot  of  Factor  Adjust  Rates. 

Adapted  from  Garrick  (2014). 


Grade 

Retirement 

Condition 

Inventory 

Condition 

Continue  Rate 

NOLH 

Factor 

Function  Rate 

E5 

nre 

neas 

96% 

96% 

92% 

nre 

eas-0 

21% 

96% 

20% 

nre 

eas-1 

87% 

96% 

84% 

re 

neas 

100% 

96% 

96% 

re 

eas-0 

67% 

96% 

64% 

re 

eas-1 

100% 

96% 

96% 

Loss  rates  are  pulled  from  the  Marine  Corps  online  Oracle  database  and  stored 
locally  within  the  MSM  program  for  the  simulation.  Local  data  tables  contain  three 
dimensions:  grade,  retirement  condition,  and  inventory  condition.  Retirement 
condition  uses  the  codes  “re”  and  “nre”  to  identify  the  parameters  of  being 
retirement  eligible  and  not  retirement  eligible,  respectively.  Inventory  conditions 
use  the  codes  “eas-0,”  “eas-1,”  and  “neas”  to  identify  a  Marine  EASing  within  the 
current  fiscal  year,  EASing  during  the  next  fiscal  year,  and  EASing  beyond  two 
years  out,  respectively. 

After  condensing  MSM’s  inputs,  a  total  of  24  factors  were  chosen  to  adjust 
the  various  inputs  of  the  MSM.  These  factors  are  annotated  in  Table  2.  The 
factors  labeled  “Loss_”  or  “Gain_”  were  factors  that  received  the  abstraction 
method.  All  other  factors  were  direct  replacements  of  MSM  inputs.  The  columns 
“Low  Value”  and  “High  Value”  indicate  the  lower  and  upper  bounds  design  points 
could  range  within.  The  column  “Decimals”  indicates  the  number  of  decimal 
placeholders  a  design  point’s  precision  was  set  to. 
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Table  2.  DOE  Factors  and  Their  Ranges.  Adapted  from  Garrick  (201 4). 


Factor  Name 

Factor  Description 

Low 

Value 

High 

Value 

Decimals 

TISmin 

Minimum  Time-In-Service  (TISmin)  required  for  a 
Marine  Agent  to  be  promoted  to  the  superior 
targeted  grade. 

-2 

3 

0 

TISmax 

Maximum  Time-In-Service  (TISmax)  allowed  for  a 
Marine  Agent  to  remain  within  its  current  grade. 

-4 

4 

0 

TIGMin 

Minimum  Time-In-Grade  (TIGmin)  required  for  a 
Marine  Agent  to  be  promoted  to  the  superior 
targeted  grade. 

-1 

3 

0 

PromTgt 

Promotion  target  (PromTgt)  is  the  desired 
cumulative  years  of  service  to  achieve  a  targeted 
grade. 

-2 

4 

1 

VIZmin 

Variable  In-Zone  (VIZ)  minimum  is  when  mean 
promotion  time  for  targeted  grade  is  within  limits, 
this  standard  selection  opportunity  is  used. 

-0.1 

0.05 

3 

VizStd 

Variable  In-Zone  (VIZ)  standard  is  when  mean 
promotion  time  for  targeted  grade  is  too  slow, 
using  VIZ  minimum  opportunity  allows  Marine 
Agents  to  compete  earlier  in  their  career,  speeding 
up  average  promotion  timing. 

-0.05 

0.05 

3 

VIZmax 

Variable  In-Zone  (VIZ)  maximum  is  when  mean 
promotion  time  for  targeted  grade  is  too  fast,  using 
VIZ  maximum  opportunity  reduces  the  number  in 
zone,  delaying  opportunity  to  compete  for  junior 
Marines  thereby  slowing  the  average  time  in 
service  for  promoted  Marines. 

-0.05 

0.1 

3 

PassO 

Passover  (PassO)  specifies  the  maximum  number 
of  passovers  a  Marine  Agent  may  have  to  remain 
within  its  current  grade. 

-2 

2 

0 

FteThsh 

Reenlistment  threshold  (ReThsh),  if  passover  limit 
is  reached,  Marine  agents  may  not  reenlist  beyond 
this  threshold. 

-2 

2 

0 

Loss_Boot 

The  loss  rate  at  which  Marine  Agents  exit  the 

Marine  Corps  during  bootcamp. 

0.6 

1.1 

3 

Loss_Retired 

The  loss  rate  at  which  Marine  Agents  exit  the 

Marine  Corps  due  to  retiring. 

0.6 

1.1 

3 

Loss_NEAS 

The  loss  rate  at  which  Marine  Agents  exit  the 

Marine  Corps  with  a  Non-EAS  separation  code. 

0.6 

1.1 

3 

Loss_Death 

The  loss  rate  at  which  Marine  Agents  exit  the 

Marine  Corps  due  to  death. 

0.6 

1.1 

3 

Loss_e2o 

The  rate  at  which  Enlisted  Marine  Agents  receive 
an  enlisted  to  officer  (e2o)  commission,  and 
therefore  leave  the  enlisted  manpower  structure. 

0.6 

1.1 

3 

Loss_Other 

The  loss  rate  at  which  Marine  Agents  exit  the 

Marine  Corps  due  to  other  means  not  documented 
with  a  specific  separation  code. 

0.6 

1.1 

3 

Loss_EAS 

The  loss  rate  at  which  Marine  Agents  exit  the 

Marine  Corps  due  to  the  end  of  their  enlisted 
contract  called  End  of  Active  Service  (EAS). 

0.6 

1.1 

3 

Loss_Reup 

The  rate  at  which  Marine  Agents  reenlist. 

0.6 

1.1 

3 

44 


Factor  Name 

Factor  Description 

Low 

Value 

High 

Value 

Decimals 

Gain_EAD 

The  rate  at  which  Marine  Agents  return  from  their 
extended  active  duty  (EAD)  recruiting  tour. 

0.6 

1.1 

3 

Gain_Desert 

Marine  Agents  that  return  after  having  become  a 
deserter. 

0.6 

1.1 

3 

Gain_Other 

Marine  Agents  that  return  to  service  for  other 
means  not  captured. 

0.6 

1.1 

3 

Gain_OCC 

The  rate  at  which  Marine  Agents  return  to  service 
with  a  OCC  gain  code. 

0.6 

1.1 

3 

Gain_Cont 

The  rate  at  which  Marine  Agents  return  to  service 
with  a  Continuous  gain  code. 

0.6 

1.1 

3 

Gain_Rever 

The  rate  at  which  Marine  Agents  return  to  service 
with  a  Reversion  gain  code. 

0.6 

1.1 

3 

Gain_Broken 

The  rate  at  which  Marine  Agents  return  from 
medical  holds. 

0.6 

1.1 

3 

After  solidifying  the  DOE  factors,  the  actual  DOE  design  was  constructed. 
We  used  the  Naval  Postgraduate  School  SEED  center’s  open  source  document, 
titled  “NOLHdesigns_v6.xls,  to  construct  our  NOLH  design  of  24  factors  and 
257  design  points  (Sanchez,  2011).  This  open  source  document  uses  MACROS 
and  hidden  formulas  within  its  template  to  automatically  compute  design  point 
values  for  the  design.  The  full  design  can  be  found  in  Appendix  E. 

To  examine  the  design’s  orthogonality,  we  constructed  a  correlations  plot 
by  using  JMP’s  multivariate  methods  functionality.  The  correlations  plot  illustrates 
how  each  NOLH  factor  relates  to  other  factors.  If  the  design  was  completely 
orthogonal,  each  comparison  would  have  a  value  of  zero.  However,  because  the 
NOLH  is  a  nearly  orthogonal  design,  some  relationships  do  exist.  From  the  24- 
factors  selected  as  inputs,  the  top  three  correlations  were: 

•  0.0876  (TISmin  with  TIGmin) 

•  0.0451  (TIGmin  with  GainOther) 

•  0.0369  (Passovers  with  LossOther) 

The  full  correlations  plot  of  the  NOLH  design  can  be  found  in  Appendix  D. 
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2.  Response  Surface  Selection 

The  MSM  produces  71  different  output  metric  categories.  In  theory,  a 
response  surface  could  be  calculated  for  each  MSM  output  metric,  but  that  level 
of  fidelity  exceeded  the  scope  of  this  thesis.  Therefore,  we  used  an  aggregation 
methodology  to  analyze  a  HRDP  process  as  a  whole  vice  drilling  into  the 
individual  subcomponents’  response  surfaces.  This  aggregated  approach  was 
applied  to  three  HRDP  process: 

•  Losses 

•  Gains 

•  Retention  Zones  A-SR 

Other  output  metrics  yielded  data  that  was  already  holistic  to  a  HRDP 
process  and  did  not  require  aggregation.  The  four  response  surfaces  that  were 
measured  with  this  approach  were: 

•  Promotions 

•  LatMoves 

•  End  Strength 

•  Patients,  Prisoners,  Transients,  and  Trainees  (P2T2) 

After  selecting  which  output  metrics  we  would  compute  response  surfaces 
for,  we  then  had  to  select  what  dimensionality  of  the  output  data  sets  we  would 
analyze.  As  described  in  Chapter  III,  MSM’s  outputs  allow  the  user  to  view 
forecasted  manpower  data  either  at  an  aggregated,  holistic  Marine  Corps,  level 
or  it  can  drill  down  into  the  MOS-Grade  level.  Additionally,  these  data  values  can 
be  viewed  at  monthly  intervals  or  at  an  aggregated  fiscal  year  level.  To  meet  the 
intent  of  this  thesis,  we  choose  the  data  set  that  was  holistic  to  the  Marine  Corps 
and  at  the  fiscal  year  interval. 

Additionally,  because  the  MSM  allows  a  user  to  specify  how  many 
forecast  years  it  will  simulate  to  and  for  how  many  repetitions,  we  had  to  isolate 
these  parameters  too.  Ultimately,  we  decided  the  fourth  forecast  year’s  data 
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output  would  be  analyzed  because  that  is  generally  the  first  opportunity  when 
enlistment  contracts  end.  As  for  repetitions,  to  yield  an  appropriate  distribution  of 
MSM’s  stochastic  capability,  it  was  decided  that  each  NOLH  design  point  would 
be  replicated  30  times.  The  average  value  from  the  replicated  data  sets  was  then 
used  for  evaluation. 

3.  Modeling  Loss  Constraints 

As  described  in  Chapter  three,  the  losses  process  uses  the  percentage 
line  non-deterministic  method  to  select  which  loss  event  a  Marine  Agent  will 
execute.  Figure  19  illustrates  this  type  of  method.  The  method  is  predicated  on 
the  fact  that  the  rates  of  all  loss  events  are  normalized  onto  the  percentage  line 
such  that  they  always  aggregate  to  a  full  100%  value.  As  a  reminder,  eight  loss 
events  and  one  continue  simulation  event  make  up  the  list  of  possible  transition 
events  a  Marine  Agent  can  execute: 

•  Bootcamp  Loss  Event 

•  Retirement  Loss  Event 

•  NEAS  Loss  Event 

•  Death  Loss  Event 

•  Enlisted  to  Officer  Commission  Loss  Event 

•  Other  Separations  Code  Loss  Event 

•  EAS  Loss  Event 

•  Reenlistment  Loss  Event 

•  Continue  Simulated  Career  Event 

Normalization  is  ensured  due  to  the  structure  of  the  supporting  data  tables 
and  how  the  computations  source  code  was  written  in  the  algorithm.  Specially, 
the  data  look-up  tables  organize  its  data  such  that  the  losses  table  contains  only 
the  data  used  for  computing  the  loss  events.  This  ensures  all  loss  events  are 
mutually  exclusive,  and  the  entire  loss-data  sampling  space  will  be  used  for 
processing  one  Marine  Agent.  This  inherent  setup  becomes  problematic  for  the 
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DOE  experiment  because  the  DOE  Factors  will  adjust  the  rates  of  the  loss 
events  independently,  and  therefore,  may  yield  DOE  design  points  where  the 
aggregation  of  loss  events  exceeds,  or  falls  short  of,  the  intended  100%  value. 
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Loss  subcategories  are  not  drawn  to  scale,  and  their  respective  values  were 
arbitrary  created  for  this  illustration.  Intent  of  this  illustration  is  to  show  how  the 
loss  subcategories  marry  up  on  a  number  line,  that  they  sum  to  100%,  and  that 
the  Mersenne  Twister  generates  a  random  “pLoss”  percentage  used  to  execute 
one  of  the  potential  subcategories. 

Figure  19.  Percentage  Line  Non-deterministic  Method. 

Adapted  from  Garrick  (2014). 


After  analyzing  the  constraint,  three  courses  of  actions  presented 
themselves  for  how  to  proceed.  The  first  option  was  to  exclude  the  Losses 
Process  from  the  NOLH  design,  and  therefore  not  evaluate  the  process.  This 
option  was  discarded  because  the  loss  function  is  one  of  the  most  important 
functions  of  the  model.  This  is  because  the  model  tries  to  maximize  its  force 
structure  by  adding  Marine  Agents  into  the  MOS-Grade  state  graph  network  but 
is  limited  by  the  constraint  that  there  must  be  available  vacancies  in  a  MOS- 
Grade  state  for  a  Marine  Agent  to  move  into  it.  Therefore,  if  Marine  Agents  do  not 
vacate  a  MOS-grade  state  the  model  becomes  stagnant. 

Option  two  was  to  include  all  possible  transition  events,  allowing  the 
NOLH  design  to  adjust  them  independently,  and  then  normalize  them  prior  to 
running  the  execution  statement  block  so  they  always  summed  to  the  full  100% 
value.  This  option  was  discarded  as  it  greatly  decreased  the  orthogonality 
between  the  factors. 
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The  third  option  was  to  focus  on  the  eight  subcomponents  that  comprise 
the  loss  category,  adjust  each  of  them  independently  as  the  NOLH  design 
dictates,  and  then  use  the  ninth  subcomponent  “Continue”  to  absorb  any 
remaining  difference  in  the  number  line  so  that  the  statement  block  still  sums  to 
100%.  While  the  “Continue”  subcomponent  would  no  longer  remain  independent, 
the  objective  of  the  experiment  would  still  be  achieved  because  all  the 
subcomponents  that  capture  loss  would  still  be  varied  independently,  and 
therefore  their  inputs  would  be  independently  captured  in  loss’s  response 
surface.  To  ensure  a  “Continue”  absorbing  event  had  available  space  to  exist  on 
the  percentage  line,  the  DOE  loss  factors  were  bounded  from  60%  to  110%  of 
their  original  baseline  values.  While  this  factor  range  focused  on  lower  rate 
values,  orthogonality  of  the  experiment  was  retained  with  this  option,  and 
therefore  still  yielded  accurate  metamodels  for  this  experiment.  This  was  the 
option  chosen  to  comply  with  the  loss  process’s  constraints. 

4.  Scaling  Constraints 

As  discussed  in  Chapter  III,  the  MSM  possesses  the  capability  for  a  user 
to  insert  optional  inputs.  These  optional  inputs  allow  a  user  to  build  a  specific 
case  study  to  be  simulated,  or  simply  allows  for  ad  hoc  simulation  adjustments. 
However,  these  optional  inputs  invoke  scaling  algorithms  that  sit  on  top  of  the 
Simulation  Phase’s  processes.  When  these  scaling  algorithms  are  invoked, 
orthogonality  of  the  DOE  is  not  maintained  since  the  values  are  being  adjusted 
independently  of  the  DOE’s  design  points  (Sanchez  &  Wan,  2012).  Therefore, 
this  DOE  refrained  from  entering  any  values  into  the  optional  inputs. 

B.  PROCEDURES 

1.  Experiment 

This  experiment  required  two  efforts  to  be  executed.  The  first  effort  was  to 

establish  the  baseline  MSM  inputs.  This  would  be  the  baseline  that  the  DOE 

factor’s  design  points  would  adjust  from.  We  set  up  the  MSM  to  simulate  four 

years,  for  30  repetitions.  For  the  reference  inputs,  we  used  the  end  of  fiscal  year 
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2013  data  to  initialize  the  MSM.  The  initial  forecast  year  was  therefore  2014.  As 
previously  annotated  in  Section  A,  no  option  inputs  were  inserted  in  order  to 
refrain  from  invoking  the  scaling  algorithms  of  the  MSM.  Table  three  illustrates 
the  inputs  inserted  into  the  “Distributions”  sheet  of  the  input  file  “siminputs.” 
Figure  20  illustrates  the  baseline  Enlisted  Career  Force  Controls  (ECFC)  used. 
Lastly,  the  “siminputs”  Excel  sheets  “Base  Pay”  and  “MOS  Data”  used  the 
configuration  data  approved  for  fiscal  year  2014,  respectively. 


Table  3.  New  Recruit  Shipping  Distribution  DOE  Inputs 


Oct 

Nov 

Dec 

Jan 

Feb 

Mar 

Apr 

May 

Jun 

Jul 

Aug 

Sep 

6.71% 

7.93% 

7.01% 

7.01% 

5.37% 

7.87% 

6.20% 

4.16% 

13.15% 

12.08% 

10.75% 

1 1 .76% 

|  Service/Grad  1 


T1S  T1S  TIG  Promotion 
Min  Max  Min  Target 


Promotion  Controls 
Promotion 

Timing  VIZ  V 
Tolerance  Min  Star 
(in  years) 


Min  Opportunity  | _ Passover  Controls _ | 

MaxTISTo 

VIZ  Minimum  Achieve  Minimum  Passover  Limit  Reenlistment  MaxTIS 


Min  Standard  Max 


Promotion 

Opportunities 


Threshold  Passover 


El 

0 

10 

0 

0 

1 

E2 

0 

10 

0 

0 

1 

E3 

0 

10 

0 

0 

1 

E4 

0 

10 

0 

1.4 

1 

E5 

2 

12 

0 

3.4 

1 

E6 

4 

22 

1 

7.9 

1  72% 

77% 

92% 

2 

ECC 

22 

E7 

6 

24 

2 

12.4 

1  67% 

72% 

87% 

2 

ECC 

21 

24 

E8 

8 

29 

3 

16.9 

1  62% 

67% 

82% 

2 

ECC 

23 

29 

E9 

10 

42 

2 

21.4 

1  57% 

62% 

77% 

2 

ECC 

42 

Figure  20.  Enlisted  Career  Force  Controls.  Adapted  from  Garrick  (2014). 


The  second  experimental  effort  was  the  DOE  experiment  itself.  To 
execute  the  DOE,  we  created  an  additional  Java  main  class.  The  purpose  of  this 
test-main  class  was  to  control  the  DOE  experiment  while  ingesting  the  next 
design  point’s  inputs,  running  the  MSM  program  for  that  design  point,  and  then 
extracting  the  associated  forecasted  data.  The  general  algorithm  used  by  the 
test-main’s  source  code  was  as  follows: 

1 .  Extract  NOLFI  design  point  inputs. 

2.  Update  MSM’s  input  document  “siminputs”  with  the  respective 
design  point  values. 
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3. 


Invoke  the  main  MSM  program  to  run  the  simulation  with  the 
updated  inputs. 

4.  Extract  and  record  the  output  results. 

5.  Repeat  with  the  next  design  point’s  inputs. 

2.  Analysis 

To  compute  statistics  from  the  DOE  experiment,  we  used  the  statistical 
software  suite  JMP  Pro  12.  Specifically,  the  software  provides  a  function  called 
Fit-Model.  With  this  function,  the  user  selects  the  dependent  and  independent 
variables  to  run  the  JMP  function.  The  dependent  variable  was  the  response 
surface  of  the  particular  metamodel  being  evaluated.  The  independent  variables 
were  the  DOE  input  factors  used  in  the  experiment.  JMP  software  uses  the  term 
“role  variable”  to  identify  the  response  surface  and  “model  effects”  to  specify  the 
relationship  of  how  the  independent  input  factors  will  be  analyzed.  Recalling  from 
Chapter  II,  a  NOLH  design  allows  for  main  effects,  two-way  interactions,  and 
quadratic  effects  to  be  examined.  Within  the  JMP  software,  the  DOE  factors  were 
used  to  construct  the  “model  effects”  by  importing  them  as  two  groups,  which 
produced  a  total  of  324  DOE  terms  to  be  analyzed: 

•  Polynomial  to  2nd  Degree  construction  was  used  to  view  the  main 
effects  and  quadratic  effects  terms  for  all  the  DOE  design  factors. 

•  Factorial  to  2nd  Degree  construction  was  used  to  view  all  the  two- 
way  interaction  terms  of  the  model’s  effects. 

Initially,  JMP’s  stepwise  regression  was  run  to  determine  which  input 
factors  were  selected  as  having  a  significant  impact  on  the  response  surface  of 
the  metamodel.  To  accomplish  this,  JMP  uses  a  Bayesian  information  criterion 
(BIC)  for  identifying  significant  terms.  The  mathematical  formula  used  to  compute 
BIC  is  shown  in  Equation  (3),  where  k  is  the  number  of  estimated  parameters  in 
the  model  and  n  is  the  number  of  observations  in  the  data  set  (“Likelihood,  AlCc, 
and  BIC,”  n.d.).  This  initial  computation  of  significant  terms  reduced  the  number 
of  terms  to  be  analyzed  by  approximately  90%. 

BIC  =  -2  log  Likelihood  +  k \n(n)  (3) 
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With  the  updated  significant  terms  selected,  JMP’s  Fit-Model  was  run  a 
second  time  using  a  Standard  Least  Squares  regression.  This  final  regression 
produced  seven  statistical  reports  that  enabled  the  analysis  of  each  metamodel: 

•  Effect  Summary:  This  report  yields  the  “LogWorth  values  for  the 
effects  in  the  model”  (“Effect  Summary  Report,”  n.d.).  The 
mathematical  formula  to  compute  LogWorth  is  shown  in  Equation 

(4) ,  where  p  is  the  term’s  p-value. 

LogWorth  =  —  log  10(p) 

•  Actual  by  Predicted  Plot:  This  plot  visually  shows  the  reader  how 
well  the  produced  metamodel  fits  the  input  data.  In  the  case  of  our 
DOE  experiment,  the  input  data  are  design  points  of  the  NOLH. 
The  actual  versus  predicted  values  are  a  comparison  of  what  the 
model  actual  yielded  during  its  stochastic  computations  compared 
to  what  the  metamodel’s  response  surface  equation  calculated. 

•  Summary  of  Fit.  This  statistical  report  provides  numerous  values 
which  yield  a  statistical  overview  of  the  model.  For  this  evaluation 
the  adjusted  R-Square  value  was  used  to  normalize  and  compare 
the  metamodels  against  one  another  because  each  model  could 
have  different  numbers  of  identified  significant  terms.  This  number 
ranges  from  0  to  1,  where  the  closer  the  value  is  to  1  the  better  the 
model’s  fit  is  to  the  data.  Its  specific  formula  is  shown  in  Equation 

(5) ;  where  MS  is  the  means  square  from  the  model’s  source  Error, 
the  source  of  the  sum  of  squares  is  the  corrected  total  (C.  Total), 
and  DF  is  the  degrees  of  freedom  from  the  model’s  corrected  total 
(“Regression  Reports,”  n.d.,  Summary  of  Fit  section). 

Rsquare  _  Aclj  =  1 - -  (5) 

SumOfSquares  /  DF 

•  Analysis  of  Variance.  This  report  yields  model  calculated  values, 
such  as  the  mean  square  for  the  metamodels  error  used  in 
Equation  (5). 

•  Parameter  Estimates.  This  report  shows  the  estimates  for  all  terms 
post  BIC  selection.  These  estimates  represent  the  coefficients  used 
in  the  response  surface  equation.  This  report  also  includes  the 
Intercept  estimate,  which  is  referring  to  the  intercept  coefficient  for 
the  response  surface  equation. 

•  Sorted  Parameter  Estimates.  This  report  shows  the  parameter 
estimates  for  each  term,  where  each  term  is  sorted  in  a  decreasing 
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order  of  significance.  Significance  is  determined  by  the  absolute 
value  of  the  calculated  “t  Ratio,”  which  is  the  ratio  of  the  estimate 
when  compared  to  its  standard  error  (“Estimates,”  n.d.,  Sorted 
Estimates  section). 

•  Prediction  Profiler.  This  report  yields  a  visual  that  allows  the  user  to 
gauge  the  sensitivity  of  the  metamodel’s  terms,  that  is  the  response 
surface’s  gradient.  As  a  quick  reference,  profiles  that  have  a  more 
horizontal  orientation  have  little  impact  on  the  metamodel’s  output 
measurements  as  the  term  changes.  Conversely,  profiles  that  have 
a  more  vertical  orientation  are  considered  to  be  sensitive  because  a 
change  in  its  value  can  cause  a  significant  change  in  the 
metamodels  output  measurements. 

Lastly,  all  the  significant  terms  from  all  the  metamodels  were  aggregated 
and  analyzed  to  determine  which  terms  had  an  overall  significant  impact  on  the 
MSM. 

C.  ANALYSIS 

This  experiment  was  computationally  intensive.  Each  design  point  took  an 
average  of  16  minutes  to  run,  utilizing  an  Intel  Core  i7  processor  with  a  16  GB 
RAM.  Extrapolating  that  timeline  over  257  design  points,  this  experiment  took 
approximately  2.85  days  to  complete. 

1.  Losses  Metamodel 

The  first  metamodel  evaluated  was  Losses.  The  response  surface  metric 
was  constructed  by  aggregating  all  the  loss  metrics  output  by  the  MSM. 
Specifically,  this  included  the  loss  categories  with  Non-EAS  and  EAS  separation 
codes.  Each  separation  code  has  the  following  sub-categories:  NEAS,  deaths, 
E20,  retired,  EAS,  bootcamp,  and  other.  Reference  Appendix  C  for  more 
information  regarding  MSM’s  output  metrics. 

Figure  21  shows  the  seven  JMP  statistical  reports  for  this  metamodel.  This 
model  had  an  excellent  fit,  indicated  by  its  0.98778  adjusted  R-Square  value. 
This  is  visually  reinforced  with  the  “Actual  by  Predicted  Plot,”  where  the  model’s 
response  surface  equation  predicted  a  forecast  that  was  within  1 .3%  of  the  actual 
simulated  output  measurements. 


53 


For  our  first  metric,  correlation,  the  metamodel  yielded  very  promising 
information.  A  visual  of  this  correlation  is  shown  in  the  Prediction  Profiler  report; 
where  it  plots  the  profile  for  the  six  factors  that  yielded  the  greatest  effects.  From 
initial  inspection,  the  six  factors  all  had  gradients  that  coincide  with  the  Marine 
Corps’  HRDP  process.  The  two  greatest  factors  were  loss  due  to  end  of  active 
service  (EAS)  and  reenlisting  (Reup).  For  EAS,  the  profile  shows  that  as  the  rate 
of  EAS  increases,  so  does  the  quantity  of  Marine  Corps  total  losses  for  a  fiscal 
year,  which  is  as  expected.  The  profile  also  indicates  an  exponential  relationship. 
Specifically,  lower  EAS  loss  rates  have  a  greater  sensitivity  in  predicting  overall 
loss  quantities. 

Conversely,  the  Reup’s  loss  rate  had  a  linear  relationship  in  the  negative 
gradient  direction.  This  directly  aligns  with  the  HRDP  because  if  more  Marines 
are  staying  in  the  Corps  for  a  subsequent  term  enlistment  then  there  are  less 
Marine  Agents  that  could  exit  the  Corps.  The  other  rates  of  non-EAS  and 
bootcamp  losses  yielded  gradients  that  had  the  correct  slope  and  magnitude, 
both  of  which  had  less  of  an  impact  on  the  over  loss  quantity. 

A  factor  that  had  an  unanticipated  significance  in  the  Losses  metamodel 
was  the  minimum  time  in  grade  (TIGmin)  factor.  As  Figure  21  shows,  this  factor 
had  both  main  effects  and  quadratic  effects  on  the  models  response  surface.  The 
Prediction  Profile  for  TIGmin  reports  that  as  the  TIGmin  increases  there  is  an 
exponential  negative  gradient  impact  on  the  losses  response  surface.  This  may 
be  an  unanticipated  affect  that  counters  the  rule  constraints  associated  with  the 
enlisted  career  force  controls  (ECFC).  Specifically,  a  Marine  Agent  can  not  be 
forced  to  exit  the  Marine  Corps  due  to  being  passed  over  for  promotion  too  many 
times  if  they  are  never  eligible  for  a  promotion  in  the  first  place. 

The  second  evaluation  metric,  input  significance  priority,  is  observed  in  the 
Sorted  Parameter  Estimates  report.  This  report  captured  15  terms  that  varied 
from  main  effects,  quadratic  effects,  and  two-way  interactions.  The  column 
labeled  “t  ratio”  was  the  measurement  metric  used  to  evaluate  input  significance. 
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Table  3,  located  at  the  end  of  this  section,  aggregates  and  compares  the  terms 
from  all  response  surfaces. 


A  -  Response  Losses 
A  Effect  Summary 


Source 

Log  Worth 

PValue 

L.EAS 

217967 

0.00000 

L.Reup 

143.064 

0.00000 

L.NEAS 

77.368 

0.00000 

TIGmin 

60.378 

0.00000 

TIGmin*TIGmin 

47.615 

0.00000 

L.EAS*L.EAS 

33.582 

. ~T 

0.00000 

L.Boot 

14.144 

1 

0.00000 

L.NEAS*L.EAS 

12.480 

mm 

0.00000 

TIGmin’L.NEAS 

6.354 

0.00000 

TISmin*L.EAS 

4.320 

n 

0.00005 

TISmin 

3.904 

] 

0.00012 

TISmin*TISmin 

3.554 

0.00028 

L.EAS”L.Reup 

3.194 

0.00064 

TIGmin*L.EAS 

2.186 

= 

0.00652 

TISmin*L.Boot 

0.044 

0.90320 

Remove  Add  Edit  I  I  FDR  denotes  effects  with  containing  effects  above  them) 


a  Actual  by  Predicted  Plot 


Losses  Predicted  P<.0001  RSq=0.99  RMSE=207.71 

Figure  21.  Losses  Model  Regression  Analysis 
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41  Summary  of  Fit 

RSquare  0.988499 

RSquareAdj  0.98778 

Root  Mean  Square  Error  207.7148 

Mean  of  Response  29403.93 

Observations  (or  Sum  Wgts)  256 

a  Analysis  of  Variance 


Sum  of 


Source  DF  Squares  Mean  Square  F  Ratio 

Model  15  889959368  59330625  1375.131 

Error  240  10354908  4314545  Prob  >  F 

C.  Total  255  900314276  <.0001* 


41  Parameter  Estimates 


Term 

Estimate 

Std  Error 

t  Ratio 

A 

Xl 

o 

Sm 

Q_ 

Intercept 

22505438 

1 54.9925 

145.20 

<.0001* 

TISmin 

-33.83698 

8.67502 

-3.90 

0.0001* 

(TISmin-0.49219)’(TISmin-0.49219) 

-22.74498 

6.166894 

-3.69 

0.0003* 

TIGmin 

-237.9692 

10.56859 

-22.52 

<•0001* 

(TIGmin-0.99609)*(TIGmin-0.99609) 

-168.1322 

9.050977 

-18.58 

<.0001* 

L.Boot 

746.37632 

89.83388 

8.31 

<.0001* 

L.NEAS 

2520.6538 

89.43311 

28.18 

<•0001* 

LEAS 

11002848 

89.53303 

122.89 

<.0001* 

(L.EAS-0.8496)*(L.EAS-08496) 

-1060146 

736.1404 

-1440 

<■0001* 

L.Reup 

-5233.99 

89.68576 

-58.36 

<.0001* 

(TISmin-0.4921 9)*(L.Boot-085081) 

7.8013236 

64.07839 

0.12 

0.9032 

(TISmin-0.4921 9)*(L.EAS-08496) 

-227.1354 

54.84371 

-4.14 

<.0001* 

(TIGmin-0.99609)*(L.NEAS-08501 1) 

-395.3276 

76.13827 

-5.19 

<•0001* 

(TIGmin-0.99609)*(L.EAS-08496) 

-227.2533 

82.80352 

-2.74 

0.0065* 

(L.NEAS-0.8501 1)*(L.EAS08496) 

-4298832 

557.4738 

-7.71 

<.0001* 

(L.EAS-0.8496)’(L.Reu  p-084937) 

-1305  53 

666.4464 

-3.46 

0.0006* 

41  Sorted  Parameter  Estimates 


Term 

Estimate 

Std  Error 

t  Ratio 

Prob>|t| 

L.EAS 

11002848 

89.53303 

122.89 

<.0001* 

L.Reup 

-5233.99 

89.68576 

-58.36 

m 

<.0001* 

L.NEAS 

2520.6538 

89.43311 

28.18 

r 

<.0001* 

TIGmin 

-237.9692 

10.56859 

-22.52 

t 

<.0001* 

(TIGmin-0.99609)*(TIGmin-0.99609) 

-168.1322 

9.050977 

-18.58 

L 

<.0001* 

(L.EAS-0.8496)*(L.EAS-08496) 

-1060146 

736.1404 

-1440 

[ 

<.0001* 

L.Boot 

746.37632 

£.9  £  :  C  SE 

8.31 

i 

<0001* 

(L.NEAS-0.8501 1)*(L.EAS-08496) 

-4298832 

557.4738 

-7.71 

<.0001* 

(TIGmin-0.99609)*(L.NEAS-08501 1) 

-395.3276 

76.13827 

-5.19 

<.0001* 

(TISmin-0.4921 9)*(L.EAS-08496) 

-227.1354 

54.84371 

-4.14 

<.0001* 

TISmin 

-33.83698 

8.67502 

-3.90 

0.0001* 

(TISmin-0.4921 9)*(TISmin-0 .49219) 

-22.74498 

6.166894 

-3.69 

0.0003* 

(L.EAS-0.8496)*(L.Reu  p-084937) 

-2305.521 

666.4464 

-3.46 

0.0006’ 

(TIGmin-0.99609)*(L.EAS-08496) 

-227.2533 

82.80352 

-2.74 

0.0065* 

(TISmin-0.4921 9)*(L.Boot-085081) 

7.8013236 

64.07839 

0.12 

0.9032 

41  - 


Prediction  Profiler 


29932.03 

[29877 8, 
299862] 


34000 
32000 
30000 
28000 
16  COO 
24000 


0.4922  0.9961  0.85081  0.85011 

TISmin  TIGmin  L.Boot  L.NEAS 


0.8496  0.84937 

L.EAS  L.Reup 


Figure  21  Continued. 


Losses  Model  Regression  Analysis 
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2. 


Gains  Metamodel 


The  second  metamodel  evaluated  was  gains.  The  response  surface 
metric  was  constructed  by  aggregating  all  the  gain  metrics  output  by  the  MSM. 
Specifically,  this  included  the  gain  sub-categories:  EAD,  Occ,  Rever,  Cont, 
Broken,  Deserter,  and  other.  Reference  Appendix  C  for  more  information 
regarding  MSM’s  output  metrics. 

Figure  22  shows  the  seven  JMP  statistical  reports  for  this  metamodel.  This 
model  had  an  excellent  fit,  indicated  by  its  0.95829  adjusted  R-Square  value. 
This  is  visually  reinforced  with  the  “Actual  by  Predicted  Plot,”  where  the  model’s 
response  surface  equation  predicted  a  forecast  that  was  within  4.2%  of  the  actual 
simulated  output  measurements. 

For  our  first  metric,  the  metamodel  yielded  excellent  correlation  with 
respect  to  the  gain  factors.  Four  out  of  the  seven  gain  factors  were  identified  as 
being  significant.  Additionally,  the  four  selected  gain  factors  all  had  positive 
gradients,  which  coincides  with  the  HRDP.  Conversely,  for  factors  of  losses  were 
determined  to  be  significant,  where  two  had  positive  gradients  and  two  had 
negative  gradients.  We  viewed  this  as  being  positive,  because  even  though  they 
were  determined  to  be  significant  their  gradient  directions  offset  each  other. 
Therefore  there  aggregated  effect  had  little  to  no  impact  on  the  metamodel.  Two 
factors  that  were  a  surprised  to  have  been  identified  as  significant  were  TISmin 
and  TIGmin. 

An  important  item  to  reiterate  is  that  the  Gains  process  is  separate  from 
the  Accessions  process.  Where  the  later  deals  predominately  with  new  recruits 
entering  the  Marine  Corps,  the  Gains  process  focuses  on  Marines  with  prior 
service  that  re-enter  the  Marine  Corps’  manpower  accountability. 
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a  »  Response  Gains 
a  Effect  Summary 


Source 

LogWorth 

G.Occ 

116.149 

G.Dstr 

108.249 

G.Oth 

74.210 

TIGmin 

48.179 

TISmin 

28.073 

L.EAS 

20.575 

G.EAD 

19.262 

TIGmin*TIGmin 

17.492 

TISmin*TIGmin 

12.079 

TISmin*TISmin 

10.596 

TIGmirTG.Dstr 

4.161 

L.Reup 

3.904 

L.NEAS 

2.587 

L.Boot*G.EAD 

1.080 

L.NEAS*L.Reup 

0.766 

G.Dstr*G.Occ 

0.441 

TISmin*L.EAS 

0.302 

L.Boot 

0.209 

TIGmin*G.Oth 

0.069 

Remove  Add 

Edit  □  FDR 

PValue 

0.00000 

0.00000 

0.00000 

0.00000 

0.00000 

0.00000 

0.00000 

0.00000 

0.00000 

0.00000 

0.00007 

0.00012 

0.00259 

0.08313 

0.17146 

0.36206 

0.49940 

0.61867 

0.85242 


A 


a  Actual  by  Predicted  Plot 


Gains  Predicted  P<.0001  RSq=0.96  RMSE=15.722 


A  Summary  of  Fit 

RSquare 

0.958291 

RSquare  Adj 

0.954933 

Root  Mean  Square  Error 

15.72186 

Mean  of  Response 

731.3281 

Observations  (or  Sum  Wgts) 

256 

a  Analysis  of  Variance 

Sum  of 

Source  DF  Squares 

Mean  Square  F  Ratio 

Model  19  1340262.7  70540.1  285.3831 

Error  236  583338  2472  Prob  >  F 

C.  Total  255  1  3985964  <.0001* 

Figure  22.  Gains  Model  Regression  Analysis 
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2i  Parameter  Estimates 


Term 

Estimate 

Std  Error 

t  Ratio 

A 

-O 

o 

oZ 

Intercept 

-65.55487 

1 6.38959 

-4.00 

<.0001* 

TISmin 

8.4038675 

0.657177 

12.79 

<•0001* 

TIGmin 

15.088175 

0.801316 

18.83 

<•0001* 

L.Boot 

-3.389651 

6.801107 

-0.50 

0.6187 

L.NEAS 

20.61325 

6.769176 

3.05 

0.0026* 

LEAS 

70.920336 

6.776908 

10.46 

<•0001* 

L.Reup 

-26.49265 

6.790213 

-3.90 

0.0001’ 

G.EAD 

68.393713 

6.810322 

10.04 

<•0001* 

G.Dstr 

276.51739 

6.770917 

4084 

<.0001* 

G.Oth 

184.89204 

6.77649 

2728 

<•0001* 

G.Occ 

303.42149 

6.810352 

44.55 

<•0001* 

(TISmin-0.49219)*(TIGmin-0 .99609) 

-4.247221 

0.561019 

-7.57 

<.0001* 

(TISmin-0.49219)*(L.EAS-08496) 

2,7941542 

4.130475 

0.68 

0.4994 

(TIGmin-0.99609)*(G.Dstr-0  84986) 

24.180282 

5.967729 

4.05 

<.0001* 

(TIGmin-0.99609)’(G.Oth-085012) 

-0.976455 

5.243185 

-0.19 

0.8524 

(L.Boot-0. 85081  )*(G.EAD-085087) 

-88.10257 

50.62888 

-1.74 

0.0831 

(L. NEAS-0. 8501 1)*(L.Reu  p-084937) 

65.042312 

47.41715 

1.37 

0.1715 

(G.Dstr-0.84986)*(G.Occ-084909) 

47.223294 

51.71095 

0.91 

0.3621 

(TISmin-0.49219)’(TISmin-0.49219) 

3.2598 

0.465246 

7.01 

<.0001* 

(TIGmin-0.99609)*(TIGmin-0 .99609) 

6.5320147 

0.690339 

9.46 

<.0001* 

Sorted  Parameter  Estimates 

Term 

Estimate 

Std  Error 

t  Ratio 

G.Occ 

303.42149 

6.810352 

44.55 

G.Dstr 

276.51739 

6.770917 

4084 

G.Oth 

184.89204 

6.77649 

2728 

TIGmin 

15.088175 

0.801316 

18.83 

TISmin 

8.4038675 

0.657177 

12.79 

L.EAS 

70.920336 

6,776908 

10.46 

G.EAD 

68.393713 

6.810322 

10.04 

(TIGmin-0.99609)’(TIGmin-0.99609) 

6.5320147 

0.690339 

9.46 

(TISmin-0.49219)*(TIGmin-0 .99609) 

-4.247221 

0.561019 

-7.57 

(TISmin-0.49219)*(TISmin-0.49219) 

3.2598 

0.465246 

7.01 

(TIGmin-0.99609)*(G.Dstr-0 84986) 

24.180282 

5.967729 

4.05 

L.Reup 

-26.49265 

6.790213 

-3.90 

L.NEAS 

20.61325 

6,769176 

3.05 

(L.Boot-0. 85081  )*(G.EAD-085087) 

-88.10257 

50.62888 

-1.74 

(L.NEAS-0.8501 1)*(L.Reup-084937) 

65.042312 

47.41715 

1.37 

(G.Dstr-0.84986)*(G.Occ-084909) 

47.223294 

51.71095 

0.91 

(TISmin-0.49219)*(L.EAS-08496) 

2,7941542 

4.130475 

0.68 

L.Boot 

-3.389651 

6.801107 

-0.50 

(TIGmin-0.99609)’(G.Oth-085012) 

-0.976455 

5.243185 

-0.19 

21  *  Prediction 


Prob>|t| 

<■0001* 

<.0001* 

<.0001* 

<•0001* 

<.0001* 

<.0001* 

<•0001* 

<.0001* 

<■0001* 

<.0001* 

<•0001* 

0.0001* 

0.0026* 

0.0831 

0.1715 

0.3621 

0.4994 

0.6187 

0.8524 


TISmin  TIGmin  L.Boot  L.NEAS  LEAS 


0.84937  0.85087  0.84986  0.85012  0.84909 


LReup  G.EAD  G.Dstr 


G.Oth 


G.Occ 


Figure  22  Continued.  Gains  Model  Regression  Analysis 
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3. 


Promotions  Metamodel 


Figure  23  depicts  Promotions  metamodel.  For  the  response  surface,  we 
used  the  direct  output  metric  promotions  in  (Promln).  The  Actual  by  Predicted 
Plot  report  showed  signs  of  banding  within  the  data  set.  Banding  is  a  term  that 
refers  to  output  measures  that  do  not  have  a  continuous  distribution.  Instead,  the 
data  groups  together  in  clusters.  This  implies  the  model  hosted  an  aspect  of 
discrete  outcomes.  To  capture  these  discrete  bands,  we  used  JMP’s  functionality 
to  create  a  partition  tree  on  the  data  set.  As  anticipated,  the  first  split  identified 
the  factor  TIGmin  as  having  a  discrete  impact  on  the  model.  A  split  categorized 
the  factor  into  leaves  <3  and  >3,  and  was  responsible  for  explaining  52%  of  the 
original  variance  in  the  model. 

With  the  Partition  Tree’s  leaves  established,  a  standard  least  squares 
regression  was  re-run,  this  time  incorporating  the  leaves  into  its  regression.  The 
recalculated  regression  showed  a  much  better  model  fit  using  the  hybrid 
regression.  The  adjusted  R-Square  value  increased  from  81%  to  94%,  justifying 
that  this  was  the  correct  model  to  explain  the  Promotions  response  surface. 

Focusing  on  the  Prediction  Profile  report,  this  model  showed  a  lot  of 
expected  behaviors.  For  the  loss  rates  of  retirement,  NEAS,  and  EAS,  all  three 
had  positive  gradients.  This  fact  supports  the  billet  availability  HRDP  constraint  in 
that  availability  must  exist  in  order  for  a  Marine  to  be  promoted  into  a  higher 
grade.  These  loss  rates  confirm  that  as  the  loss  rate  increases,  that  is  more 
Marines  are  exiting  the  Corps,  the  more  billets  become  available  for  junior 
Marines  to  promote  into.  Additionally,  the  factor  TISmin  yielded  a  profile  that 
supported  the  FIRDP.  Specifically,  if  the  ECFC  constraints  require  a  greater 
TISmin  for  a  particular  billet,  that  ECFC  is  therefore  decreasing  the  pool  of 
possible  candidates  that  could  be  considered  for  promotion. 
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A  Response  Promotions 
A  Effect  Summary 


Source 

LogWorth 

Leaf  Label 

125.766  [ 

L.EAS 

100.761  F 

L.Reup 

42.895  F 

TISmin 

1 6.662  L 

TISmin*TISmin 

1 2.832  [ 

L.NEAS 

6.566  [ 

L.Ret 

5.916  [ 

TISmin’L.EAS 

5.095  F 

L.EAS’L.EAS 

3.177  f 

L.EAS’L.Reup 

0.833  I 

L.Ret*L.NEAS 

0.392  | 

Remove  Add  Edit  FDR 

PValue 

0.00000 

0.00000 

0.00000 

0.00000 

0.00000 

0.00000 

0.00000 

0.00001 

0.00066 

0.14702 

0.40558 


a  Actual  by  Predicted  Plot 


Promotions  Predicted  P  <.0001  RSq=0.95 
RMSE=3018.3 


^Summary  of  Fit 

RSquare  0.945581 

RSquare  Adj  0.943127 

Root  Mean  Square  Error  301 8.32 

Mean  of  Response  65887.09 

Observations  (or  Sum  Wgts)  256 

A  Analysis  of  Variance 

Sum  of 

Source  DF  Squares  Mean  Square  F  Ratio 

Model  11  3.8625e*10  3.5113e<9  385.4272 

Error  244  2222902164  91102548  Prob  >  F 

C  Total  255  4.0848e*10  <.0001* 


Figure  23.  Promotions  Model  Regression  Analysis 
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a  Parameter  Estimates 


Term 

Estimate 

Std  Error 

t  Ratio 

Prob>  |t| 

Intercept 

25178424 

2247.606 

11.20 

<.0001* 

Leaf  Label[TIGmin<3] 

14050.121 

292.0981 

48.10 

<.0001* 

TlSmin 

-1154.38 

126.0185 

-9.16 

<.0001* 

L.Ret 

6489.7881 

1303.65 

4.98 

<.0001* 

LNEAS 

6875.724 

1299.715 

5.29 

<.0001* 

LEAS 

47861.152 

1301.713 

36.77 

<.0001* 

L.Reup 

-22296.19 

1303.962 

-17.10 

<■0001* 

[TISmin-0.4921 9)*(L.EAS-08496) 

-3619.702 

793.4596 

-4.56 

<.0001* 

(L.Ret-0.85035)*(L.NEAS-08501 1) 

-1019343 

12234.94 

-0.83 

0.4056 

(L.EAS-0.8496)*(L.Reup-0.84937) 

-17311.74 

11899.92 

-1.45 

0.1470 

(TISmin-0.49219)*(TISmin-0.49219) 

-704.5717 

89.96433 

-7.83 

<■0001* 

(L.EAS-0.8496)*(L.EAS-08496) 

-3647929 

10579.58 

-3.45 

0.0007* 

A  Sorted  Parameter  Estimates 

Term 

Estimate 

Std  Error 

t  Ratio 

Leaf  Label[TIGmin<3] 

LEAS 

L.Reup 

TlSmin 

(TISmin-0.49219)*(TISmin-0 .49219) 

LNEAS 

LRet 

(TISmin-0.4921 9)*(L.EAS-0£496) 
(L.EAS-0.8496)*(L.EAS-08496) 
(L.EAS-0.8496)*(L.Reup-0.84937) 
(L.Ret-0.8503  5)*(L.NEAS-08501 1) 

A  r  Prediction  Profiler 


.2  71749.19 
|  [71087.5, 
2  72410.9] 


90000 
S  j  000 
70000 
60000 
50000 
40000 


Prob>|t| 

<.0001* 

<.0001* 

<•0001* 

<.0001* 

<■0001* 

<•0001* 

<■0001* 

<.0001* 

0.0007* 

0.1470 

0.4056 


TIGmin<3 
TIGmin<3 
Leaf  Label 


0.4922 

TlSmin 


oo  m  r- 

o  o  o  ci 
0.85035 
LRet 


O  N  CO  ffl  r 

o  o  o  d 
0.85011 
LNEAS 


1C  MO  fft  r  c  «  N  ffl  Ol  r-  f 

o  o  o  o 

0.8496 


•-  o  o  o  o  — 


L.EAS 


0.84937 

L.Reup 


Figure  23  Continued.  Promotions  Model  Regression  Analysis 

4.  LatMoves  Metamodel 


Figure  24  depicts  lateral  moves  (LatMoves)  metamodel.  For  the  response 
surface,  we  used  the  direct  output  metric  lateral  moves  in  (LMIn).  The  LatMoves 
model  returned  the  second  best  Summary  of  Fit  report  characteristics  when 
compared  to  the  other  metamodels.  Specifically,  it  calculated  an  impressive 
adjusted  RSquare  value  of  99%,  meaning  an  almost  exact  fit  between  the 
metamodel  and  the  existing  data. 

With  respect  to  correlation  and  gradients,  the  LatMoves  metamodel 
closely  mimicked  the  Promotions  metamodel.  This  makes  sense  because  while 
promotions  analysis  is  concerned  with  the  vertical  accession  through  the  MOS- 
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Grade  state  graph  network,  LatMoves  is  concerned  with  the  horizontal 
transitions.  In  both  cases,  vacancies  must  exist  to  facilitate  the  need  for  a  Marine 
to  LatMove  into  that  identified  MOS. 

With  respect  to  term  significance,  loss  rates  once  again  dominated  the 
metamodel.  This  again  reinforces  the  fact  that  the  MSM  is  built  around  the  billet 
vacancy  constraint;  that  vacancies  must  exist  in  order  to  allow  Marine  Agents  to 
move  around  within  the  MOS-Grade  state  graph  network. 


d  »  Response  LatMoves 
4  Effect  Summary 


Source 

LEAS 

l.Reup 

L.NEAS 

L.Ret 

L.e2o 

LEAS'LReup 

TIGmin'LRet 

6-Occ 

TIGmin'LEAS 

LReup'LReup 

TKjmin’LNEAS 

L-Ret'GDstr 

TlGfn»n*TK3nvi 

L.Ret' L.Reup 

Le2o'6.0cc 

l.  Ret' L.NEAS 

TIGmin 

GDstr 


Log  Worth 

224.212 
178.190 
94.489 
77.442 
20.030 
19.122 
9.195 
5.371 
5.369 
4.674 
3.746 
3.190 
2424 
2.387 
0  515 
0.248 
0.209 
0.002 


PValue 

0.00000 

0.00000 

0,00000 

0.00000 

0,00000 

0.00000 

0.00000 

0.00000 

0.00000 

0.00002 

0.00018 

0.00065 

0.00376 

0.00111 

0.28482 

0.56547 

0.61822 

0.99507 


4  Actual  by  Predicted  Plot 


LatMoves  Predicted  P <.0001  RSqr0.99 
RMSE=9.0101 


4  Summary  of  Fit 

RSquar*  0991452 

RSquere  AdJ  0  990803 

Root  Mean  Square  Error  9.010139 

Mean  of  Response  583  3555 

Observations  (or  Sum  Wgts)  2  56 


Figure  24.  LatMoves  Model  Regression  Analysis 
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i  Analysis  of  Variance 

Sum  of 

Source 

DF 

Squares 

Mean  Square 

F  Ratio 

Model 

18 

2231871.1 

123993 

15  27.231 

Error 

237 

19241.6 

81 

Prob  >  F 

C.  Total 

255 

2251112.7 

<■0001* 

Parameter  Estimates 

Term 

Estimate 

Std  Error  t  Ratio  Prob>|t| 

Intercept 

190.41111 

8.842652 

21.53 

<.0001* 

TIGmin 

0.2287258 

0.458333 

0.50 

0.6182 

L.Ret 

110.21021 

3.884304 

2837 

<■0001* 

L.NEAS 

135.16122 

3  £79582 

3434 

<.ooor 

L.e2o 

39.957102 

3.884513 

10.29 

<.0001* 

LEAS 

518.03018 

3.883523 

133.39 

<.0001* 

L.Reup 

-328.6002 

3.889821 

-8448 

<.0001* 

G.Dstr 

-0.024025 

3.880185 

-0.01 

0.9951 

G.Occ 

-18.36313 

3.900388 

-4.71 

<.ooor 

(TIGmin-0.99609)*(L.Ret-085035) 

21 .866344 

3.392504 

6.45 

<0001* 

(TIGmin-0.99609)*(L.NEAS-08501 1) 

13.31528 

3.497996 

3.81 

0.0002* 

(TIGmin-0.99609)*(L.EAS-08496) 

17.382537 

3.692894 

4.71 

<.0001* 

(L.Ret-0.85035)’(L.NEAS-085011) 

22.141672 

38.47048 

0.58 

0.5655 

(L.Ret-0.85035)*(L.Reup-084937) 

-77.1 1275 

26.60797 

-2.90 

0.0041* 

(L.Ret-0.85035)’(G.Dstr-0.84986) 

90.320057 

26.12011 

3.46 

0.0006* 

(L.e2o-0.84954)*(G.Occ-084909) 

31.709231 

29.58031 

1.07 

0.2848 

(L.EAS-0.8496)*(L.Reup-084937) 

-359.4333 

35.96777 

-9.99 

<.0001* 

(TIGmin-0.99609)*(TIGmin-0 .99609) 

1.1582711 

0.395816 

2.93 

0.0038* 

(L.Reup-0.84937)*(L.Reup-084937) 

136.10863 

31.36948 

4.34 

<.0001* 

Sorted  Parameter  Estimates 

Term 

Estimate 

Std  Error 

t  Ratio 

Prob>|t| 

L.EAS 

518.03018 

3.883523 

133.39 

M  i  i 

<.0001* 

L.Reup 

-328.6002 

3.889821 

-8448 

:  1  1  1 

<.0001* 

L.NEAS 

135.16122 

3.879582 

3484 

<.0001* 

L.Ret 

110.21021 

3.884304 

2837 

<.0001* 

L.e2o 

39.957102 

3.884513 

10.29 

<.0001* 

(L.EAS-0.8496)*(L.Reu  p-034937) 

-359.4333 

35.96777 

-9.99 

[ 

<■0001* 

(TIGmin-0.99609)*(L.Ret-085035) 

21 .866344 

3.392504 

6.45 

<.0001* 

G.Occ 

-18.36313 

3.900388 

-4.71 

<.0001* 

(TIGmin-0.99609)*(L.EAS-08496) 

17.382537 

3.692894 

4.71 

<.0001* 

(LReup-0.84937)*{L.Reu  p-084937) 

136.10863 

31.36948 

4.34 

<.0001* 

(TIGmin-0.99609)*(L.NEAS-08501 1) 

13.31528 

3.497996 

3.81 

0.0002* 

(L.Ret-0.85035)*(G.Dstr-084986) 

90.320057 

26.12011 

3.46 

0.0006* 

(TIGmin-0.99609)*(TIGmin-0 .99609) 

1.1582711 

0.395816 

2.93 

0.0038* 

(L.Ret-0.85035)*(L.Reup-084937) 

-77.11275 

26.60797 

-2.90 

0.0041* 

(Le2o-0.84954)’(G.Occ-084909) 

31.709231 

29.58031 

1.07 

0.2848 

(L.Ret-0.85035)*(L.NEAS-08501 1) 

22.141672 

38.47048 

0.58 

0.5655 

TIGmin 

0.2287258 

0.458333 

0.50 

0.6182 

G.Dstr 

-0.024025 

3.880185 

-0.01 

0.9951 

•»  Prediction  Profiler 

„  800 
|  578.6062 
|  [576.455  650 
3  580.757]  5Q0 


«-  o  «—  <M  m 


0.9961 

TIGmin 


VD  N  CO  CTl  r- 

o  o  o  d 

0.85035 

L.Ret 


ID  N  CO  O'  >- 

d  d  d  d 
0.85011 
L.NEAS 


VO  N  00  W  «- 

d  d  d  d 
0.84954 
L.e2o 


2  578.6062 

|  [576.455, 

3  580.757] 


LEAS 


L.Reup 


G.Dstr 


G.Occ 


Figure  24  Continued.  LatMoves  Model  Regression  Analysis 
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5.  End  Strength  Metamodel 

Figure  25  depicts  the  End  Strength  metamodel.  For  the  response  surface, 
we  used  the  direct  output  metric  “End”;  refer  to  Appendix  C  to  view  all  MSM 
outputs.  The  End  Strength  model  yielded  the  worst  Summary  of  Fit  report 
characteristics.  However,  this  was  expected.  Let  us  first  explain  the  reports 
produced  by  JMP  and  then  we  will  justify  why  it  was  expected. 

From  the  initial  standard  least  squares  regression,  banding  was  identified 
which  suggested  a  factor  was  exhibiting  discrete  impact  properties.  This  was 
confirmed  with  a  Partition  Tree.  Specifically,  the  factor  TIGmin  was  again 
identified  in  having  the  same  categorical  impacts  of  <3  and  >3  leaf  values. 
Applying  a  singular  split  explained  86%  of  the  variance  within  the  model. 

With  the  Partition  Tree’s  leaves  established,  a  standard  least  squares 
regression  was  re-run,  this  time  incorporating  the  leaves  into  its  regression.  The 
recalculated  regression  showed  a  much  better  model  fit  using  the  hybrid 
regression.  The  Adjusted  RSquared  value  increased  from  72%  to  92%,  justifying 
that  this  was  the  correct  metamodel  to  explain  the  End  Strength  response 
surface. 


^  ~  Response  End  Strength 
J  Effect  Summary 


Source 


LogWorth 


PValue 


Leaf  Label 
L.EAS 

TIGmin*L.EAS 

TISmin’LEAS 

TISmin*TISmin 

TISmin 

TIGmin'L.NEAS 

L.NEAS 

TIGmin 


111.310 

12.940 

11.919 

9.068 

7.571 

4.817 

2.752 

1.185 

0.281 


0.00000 
0.00000 
0.00000 
0.00000 
0.0  0000 
0.00002  A 
0.00177 
0.06524  A 
0.52357  A 


Remove  Add  Edit  1  I  FDR  CA'  denotes  effects  with  containing  effects  above  them) 


Figure  25.  End  Strength  Model  Regression  Analysis 
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a  Actual  by  Predicted  Plot 


RMSE=2066 


A  Summary  of  Fit 

RSquare  0.925654 

RSquare  Adj  0.922934 

Root  Mean  Square  Error  2066.011 

Mean  of  Response  151813.6 

Observations  (or  Sum  Wgts)  256 


A  Analysis  of  Variance 
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Figure  25  Continued.  End  Strength  Model  Regression  Analysis 
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As  previously  stated,  the  model’s  poor  fit  characteristics  were  expected. 
This  is  due  to  the  constraint  applied  by  the  MSM  that  vacancies  must  exist  to  get 
Marine  Agents  to  move  within  the  model.  However,  the  overall  quantity  of 
vacancies  is  capped  by  the  input  GAR  force  structure.  For  this  experiment,  the 
same  GAR  structure  progression  was  used  for  every  DOE  design  point.  This 
ensured  that  every  design  point  simulation  run  had  a  fourth-year  GAR  end 
strength  input  value  of  154,836  personnel. 

Figure  26  shows  the  End  Strength  output  measurement’s  distribution. 
Specifically  it  shows  a  histogram  of  End  Strength  values  for  each  of  the 
256  design  points,  as  well  as  a  statistical  summary  for  that  histogram.  The 
produced  histogram  visually  shows  that  an  overwhelmingly  majority  of  the  256 
design  points  yielded  End  Strength  calculations  that  were  tightly  coupled  next  to 
the  GAR’s  fourth  forecasted  year.  Secondly,  the  Summary  Statistics  report 
shows  the  statistical  mean  calculated  from  this  distribution,  and  the  respective 
95%  confidence  interval  that  is  tightly  coupled  around  the  mean. 

To  summarize,  these  facts  prove  that  the  MSM  was  built  to  align  with 
future  GARs.  Therefore,  GARs  are  without  a  doubt  the  most  crucial  and 
significant  input  for  the  simulation.  It  also  yields  warning  that  the  data  output 
metrics  are  directly  dependent  on  the  GAR.  Meaning  the  manpower  forecasts 
produced  from  adjusting  any  of  the  any  internal  constraints,  like  ECFCs,  are  only 
accurate  for  the  specific  GAR  it  ran  its  simulation  off  of. 
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Figure  26.  DOE  End  Strength  Distribution  Analysis 


6.  P2T2  Metamodel 

Figure  27  depicts  the  Patients,  Prisoners,  Transients,  and  Trainees  (P2T2) 
metamodel.  For  the  response  surface,  we  used  the  direct  output  metric  “P2T2”; 
refer  to  Appendix  C  to  view  all  MSM  outputs.  Immediate  signs  with  the  P2T2 
model  suggested  that  discrete  factors  were  having  an  impact  on  its  response,  so 
a  partition  tree  was  used  to  identify  the  data  bandings  that  were  occurring.  Once 
again  TIGmin  was  identified  as  having  a  discrete  impact  on  the  model.  A  split 
categorized  the  factor  into  leaves  <3  and  >3,  and  was  responsible  for  explaining 
86%  of  the  original  variance  in  the  model. 

With  the  Partition  Tree’s  leaves  established,  a  standard  least  squares 
regression  was  re-run,  this  time  incorporating  the  leaves  into  its  regression.  The 
recalculated  regression  showed  a  much  better  model  fit  using  the  hybrid 
regression.  The  adjusted  RSquare  value  increased  from  74%  to  95%,  justifying 
that  this  was  the  correct  metamodel  to  explain  the  Gains  response  surface. 

It  was  originally  anticipated  that  the  input  significance  priority  would  have 
been  more  equally  distributed  across  a  larger  variety  of  factor  terms.  As 
explained  in  Chapter  III,  the  P2T2  is  calculated  from  the  instantiation  of  a  Marine 
Agent  being  selected  for  a  LatMove.  Therefore,  if  the  quantity  of  LatMoves 
increases,  so  should  the  quantity  of  P2T2.  Some  of  this  does  show  through  in 
Figure  27  with  respect  to  the  positive  correlation  value  for  EAS  and  the  negative 
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correlation  value  with  Reenlistment  (Reup).  Specifically,  as  these  two  factor 
terms  increase  in  positive  and  negative  values,  respectively,  more  vacancies 
would  open  up  in  MOS-Grade  states  which  therefore  would  give  more  Marines 
the  opportunity  to  LatMove  into  a  new  MOS. 


a  ■»  Response  PT2P 
A  Effect  Summary 
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6.502  Q 

L.EAS*L.EAS 

3.127  0 

PrmTgt*G.EAD 

2.444  0 

TISmin*PrmTgt 

2.418  0 

L.NEAS 

2.263  0 

TIGmin*L.NEAS 

2.178  0 

G.EAD 

2.063  0 

PrmTgt 

1.292  1 

TIGmin 

0.361  Q 

Remove  Add  Edit  I  I  FDR  CA'  denotes  effects  with  containing  effects  above  them) 


a  Actual  by  Predicted  Plot 


i  Summary  of  Fit 
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RSquare  Adj 
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Root  Mean  Square  Error 
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Analysis  of  Variance 

Source 

Sum  of 
DF  Squares 

Mean  Square 

F  Ratio 

Model 

1 5  3723750856 
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349.1426 

Error 
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Prob  >  F 

C.  Total 
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Figure  27.  P2T2  Model  Regression  Analysis 
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A  Parameter  Estimates 
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A  Sorted  Parameter  Estimates 
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Figure  27  Continued.  P2T2  Model  Regression  Analysis 


7.  Retention  Metamodel 


Figure  28  depicts  Retention  metamodel.  The  response  surface  metric  was 
constructed  by  aggregating  all  the  “In-Year,”  “Out-of-Year,”  and  “Not  Accounted 
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for”  subcategories  of  MSM’s  retention  metrics.  Reference  Appendix  C  for  more 
information  regarding  MSM’s  output  metrics. 

The  Retention  model  returned  the  best  Summary  of  Fit  characteristics  of 
any  metamodel.  Specifically,  it  boasted  an  impressive  adjusted  RSquare  value  of 
99.9%.  Additionally,  the  significant  factor  terms  had  correlation  signs  that  aligned 
with  the  HRDP  process.  Specifically,  the  EAS  loss  rate  had  a  predominate  effect 
on  the  response  surface  of  the  model,  which  is  expected  because  if  Marines 
reach  their  EAS  and  leave  the  Marine  Corps  there  will  be  a  dramatic  drop  in  the 
candidate  population  that  could  reenlist. 
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Figure  28.  Reenlistment  Model  Regression  Analysis 
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A  Parameter  Estimates 
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Figure  28  Continued.  Reenlistment 


Model  Regression  Analysis 
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8. 


Accessions  Metamodel 


Figure  29  depicts  Accessions  metamodel.  For  the  response  surface,  we 
used  the  direct  output  metric  non-prior  service  (NPS)  accessions.  Immediate 
signs  with  the  Accessions  metamodel  suggested  that  discrete  factors  were 
having  an  impact  on  its  response,  so  a  partition  tree  was  used  to  identify  the  data 
bandings  that  were  occurring.  Once  again  TIGmin  was  identified  as  having  a 
discrete  impact  on  the  model.  A  split  categorized  the  factor  into  leaves  <3  and  > 
3,  and  was  responsible  for  explaining  85%  of  the  original  variance  in  the  model. 

With  the  Partition  Tree’s  leaves  established,  a  standard  least  squares 
regression  was  re-run,  this  time  incorporating  the  leaves  into  its  regression.  The 
recalculated  regression  showed  a  much  better  model  fit  using  the  hybrid 
regression.  The  adjusted  RSquare  value  increased  from  75%  to  95%,  justifying 
that  this  was  the  correct  metamodel  to  explain  the  Gains  response  surface. 
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Figure  29.  Accessions  Model  Regression  Analysis 
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Figure  29  Continued.  Accessions  Model  Regression  Analysis 
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a  Sorted  Parameter  Estimates 
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Figure  29  Continued.  Accessions  Mode!  Regression  Analysis 

9.  Effect  Significance 

The  purpose  of  this  section  was  to  analyze  the  significant  terms  calculated 
by  each  metamodel.  The  metric  for  measuring  this  was  the  calculated  “t  Ratio,” 
which  is  viewed  in  the  Sorted  Parameter  Estimates  report,  “t  Ratio”  is  the  ratio  of 
the  estimate  when  compared  to  its  standard  error  (“Estimates,”  n.d.,  Sorted 
Estimates  section).  The  final  product  of  this  analysis  was  Table  4. 


While  it  can  be  positive  or  negative  in  order  to  identify  the  term’s  gradient 
direction,  our  analysis  chose  to  focus  on  overall  effect  significance  vice  gradient. 
To  accomplish  this,  we  took  the  absolute  value  of  each  term’s  value,  aggregated 
and  averaged  that  term,  where  blank  cells  received  a  value  of  zero,  and  then 

sorted  the  terms  in  descending  order.  Blank  cells  in  Table  4  represent  instances 
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where  that  term  was  not  identified  as  being  significant  for  that  particular 
metamodel.  Main  effects  factors  are  bolded  for  easy  viewing. 


Table  4.  Aggregated  Effects  and  their  Priority  of  Significance 
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10.  Database  Pull  Inputs 

During  the  course  of  this  experiment,  a  pattern  emerged  with  respect  to 
the  database  pulls  from  Total  Force  Data  Warehouse  (TFDW),  invoked  by 
MSM’s  source  code,  and  how  they  were  utilized  as  inputs  for  the  MSM.  While  we 
were  writing  and  debugging  our  test-main  Java  class,  the  class  that  would  run 
the  DOE  experiment,  it  was  observed  that  the  TFDW  data  puli  “Reenlistments” 
had  zero  effect  on  any  of  the  output  measurements.  Examining  this  further,  it  was 
determined  that  while  MSM  pulls  “Reenlistments”  data  from  TFDW  and  stores  it 
as  an  object  within  the  MSM,  the  MSM  never  again  references  or  invokes  this 
data.  While  the  level  of  significance  for  the  other  three  database  pulls  was  not 
examined  nor  determined,  due  to  how  this  experiment  was  setup  and  run,  the 
after-action  knowledge  gained  for  this  insignificant  impact  was  worth  annotating 
in  the  analysis  of  the  DOE  experiment 
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V.  MSM  VALIDATION 


This  chapter  examines  the  accuracy  of  the  model  with  respect  to  its  ability 
to  forecast  into  the  future.  To  complete  this  evaluation,  historical  data  from  the 
past  12  fiscal  years  was  obtained.  This  data  served  two  key  requirements  in 
order  to  complete  the  validation  study.  First,  a  specific  fiscal  year’s  data  served 
as  the  various  inputs  that  are  required  to  initialize  the  MSM.  As  discussed  in 
Chapter  III,  this  required  extracting  a  specific  fiscal  year’s  GAR  force  structure, 
initial  inventory,  all  the  reference  inputs  that  specify  the  manpower  pyramid 
transition  rates,  and  ECFCs. 

Secondly,  the  data  was  used  to  compare  MSM’s  output  measurements 
against  the  Marine  Corps’  historic  manpower  quantities.  This  comparison  was 
made  at  each  fiscal  year  iteration  within  the  MSM.  It  was  decided  to  have  each 
simulation  forecast  out  to  five  years  per  simulation  run  as  the  majority  of  enlisted 
contracts  conclude  by  the  five-year  mark,  and  therefore  would  get  one  complete 
cycle  of  Marine  Agents  through  the  program  (Garrick,  2014).  As  Figure  30 
shows,  each  simulation  run  was  iterated  such  that  the  previous  run’s  first 
forecasted  year  became  the  MSM’s  inputs. 

The  metric  for  measurement  during  this  validation  was  the  computed 
relative  error  for  the  various  categories  of  each  forecasted  year.  Equation  (6) 
depicts  the  formula  for  calculating  this  error  were  X  serves  as  the  measured 
mean  for  a  particular  category,  and  r  is  its  actual  historical  value  (Law  &  Kelton, 
2000). 


r 

Using  relative  error  allowed  for  the  formula  to  naturally  normalize  a  data 
set  to  its  respective  fiscal  year  target.  This  was  crucial  as  the  numerous 
manpower  historic  quantities  naturally  fluctuate  year  to  year.  The  equation  also 
standardized  the  calculations;  negative  calculations  indicated  the  MSM 
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underestimated  that  metric  while  positive  calculations  indicated  the  MSM 
overestimated. 
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Figure  30.  Scope  of  Validation 

It  was  decided  to  seed  the  Mersenne  Twister’s  Random  number 
Generator  with  the  sequence  224.  Seeding  the  random  number  generator 
ensures  that  each  simulation  will  follow  the  exact  same  random  number 
progression.  This  action  removes  any  dependence  on  the  random  number 
generator,  and  therefore  enabled  us  to  compare  MSM’s  forecasted  years  per 
simulation  run  amongst  each  other.  To  ensure  an  appropriate  distribution  of 
output  results,  30  repetitions  were  conducted  per  each  simulation  run. 
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A. 


UNCONTROLLABLE  FACTORS 


It  is  critical  to  understand  that  this  validation  experiment  measured  how 
the  MSM  deviates  from  historical  manpower  progressions.  However,  the  derived 
accuracy  of  this  validation  experiment  is  based  off  of  the  assumption  that  the 
historical  manpower  progression  is  constant,  and  therefore  is  controllable.  This 
unfortunately  does  not  marry  up  ideally  with  Marine  Corps’  history.  It  should 
come  to  no  surprise  that  numerous  uncontrollable  factors  have  occurred 
throughout  out  the  range  of  the  fiscal  years  that  were  selected  for  this  validation. 
Events  such  as  Operating  Iraqi  Freedom  (OIF),  Operating  Enduring  Freedom 
(OEF),  sequestration,  drawing  down  the  Department  of  Defense’s  manpower 
force  structure,  as  well  as  natural  changes  in  command  and  leadership  all 
influence  the  direction  the  Marine  Corps  intends  to  go  and  the  manpower 
required  to  accomplish  it.  This  is  being  stated  because  the  MSM’s  ability  to 
accurately  forecast  is  dependent  on  the  future  inventory  structure  GARs  that 
serve  as  its  inputs,  as  well  as  the  transition  rates  it  uses  for  estimating  Marine 
Agent  movement.  Therefore,  while  it  is  reasonable  to  assume  that  each 
successive  forecasted  year  is  anticipated  to  experience  larger  relative  error, 
external  real-life  events  have  impacted  the  Marine  Corps  to  alter  its  previously 
planned  manpower  force  structure. 

B.  SOURCE  CODE  CONSTRAINTS 

During  the  course  of  the  validation  experiment,  we  identified  that  the 
retention  zones  that  are  hard  coded  into  the  MSM’s  source  code  do  not  align 
exactly  with  current  Marine  Corps  policy.  Specifically,  MSM  defines  these  zones 
as: 

•  Zone  A:  less  than  5  years  of  service 

•  Zone  B:  less  than  9  years  of  service. 

•  Zone  C:  less  than  13  years  of  service. 

•  Zone  D:  less  than  17  years  of  service. 

•  Senior  Enlisted  Zone:  greater  than  or  equal  to  20  years  of  service. 
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Current  Marine  Corps  policy  defines  these  zones  as: 

•  Zone  A:  less  than  6  years  of  service 

•  Zone  B:  less  than  1 1  years  of  service. 

•  Zone  C:  less  than  15  years  of  service. 

•  Zone  D:  less  than  19  years  of  service. 

•  Senior  Enlisted  Zone:  greater  than  or  equal  to  1 9  years  of  service. 

Because  we  were  unable  to  determine  if  these  were  tuning  parameters 
that  were  set  by  the  developer  to  manually  adjust  the  MSM  in  hopes  of  producing 
more  accurate  results,  we  decided  to  leave  these  as  is  in  order  to  effectively 
evaluate  the  MSM  program  that  is  currently  being  used  by  M&RA. 

C.  RESULTS 

Maintaining  consistency  with  Chapter  IV’s  Design  of  Experiment,  the 
following  categories  were  examined:  aggregated  losses,  aggregated  gains, 
promotions  to  the  effect  of  drilling  down  into  promoting-to  grade  data  fields,  end 
strength,  and  retention  to  the  effect  of  drilling  down  into  retention  zone  data 
fields.  To  normalize  how  each  validation  result  was  graphed,  common 
procedures  were  enacted.  Each  simulation  run  was  labeled  by  the  first  year  it 
forecasted.  For  example  if  the  user  loaded  2005’s  inventory  and  transition  rate 
data  into  the  MSM,  with  2006’s  GAR  inventory  structure  and  Enlisted  Career 
Force  Controls,  fiscal  year  2006  would  be  the  first  forecasted  year  and  therefore 
would  also  be  the  label  name  for  that  particular  simulation  run.  When  plotting 
each  categories  relative  error  results,  it  was  decided  to  arrange  the  data  such 
that  numerical  forecasted  years  served  as  the  independent  variable  on  the  y-axis 
and  the  relative  errors  served  as  the  dependent  variable  on  the  x-axis.  This  data 
arrangement  facilitated  the  ability  to  overlay  the  simulation  runs  in  order  to 
determine  common  patterns  of  progression  within  the  MSM. 

Figure  31  through  Figure  51  are  the  relative  error  graphs  for  each 
validation  category.  To  reduce  the  clutter  of  the  graphs,  as  well  as  focus  on  time 
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Relative  Error 


periods  that  are  less  centered  on  OIF  and  OEF,  the  graphs  show  only  simulation 
runs  stating  at  fiscal  year  2009  and  beyond.  The  full  relative  error  results  can  be 
viewed  in  matrix  form  within  Appendix  G. 
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Points  were  calculated  with  Equation  6.  X  was  the  sum  of  the  outputs  Neas.n 
and  Neas.e  for  the  targeted  year,  rwas  the  quantity  of  Marines  pulled  from 
TFDW  with  the  separation  code  NEAS  for  the  targeted  year. 

Figure  31 .  NEAS  Losses  Relative  Error 
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Points  were  calculated  with  Equation  6.  X  was  the  sum  of  the  outputs  Retired. n 
and  Retired. e  for  the  targeted  year,  rwas  the  quantity  of  Marines  pulled  from 
TFDW  with  the  separation  code  Retired  for  the  targeted  year. 

Figure  32.  Retired  Losses  Relative  Error 
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Points  were  calculated  with  Equation  6.  X  was  the  sum  of  the  outputs  Other.n 
and  Other. e  for  the  targeted  year,  rwas  the  quantity  of  Marines  pulled  from 
TFDW  with  the  separation  code  Other  for  the  targeted  year. 

Figure  33.  Other  Losses  Relative  Error 
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Points  were  calculated  with  Equation  6.  X  was  the  sum  of  the  outputs  Eas.n 
and  Eas.e  for  the  targeted  year,  r  was  the  quantity  of  Marines  pulled  from  TFDW 
with  the  separation  code  EAS  for  the  targeted  year. 

Figure  34.  EAS  Losses  Relative  Error 
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Points  were  calculated  with  Equation  6.  X  was  the  sum  of  the  outputs  Ead.g, 
Occ.g,  Rev.g,  Cont.g,  Broke. g,  Des.g,  and  Other.g  for  the  targeted  year,  r  was 
the  quantity  of  Marines  pulled  from  TFDW  with  the  same  gain  codes  for  that  year. 

Figure  35.  Aggregated  Gains  Relative  Error 

85 


—♦—2009 
-■-2010 
— * — 2011 

- 2012 

—♦—2013 

-•-2014 


Points  were  calculated  with  Equation  6.  X  was  the  output  NPS.g  for  the 
targeted  year,  rwas  the  quantity  of  Marines  pulled  from  TFDW  for  total 
accessions  for  the  targeted  year. 

Figure  36.  Accession  Gains  Relative  Error 
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Points  were  calculated  with  Equation  6.  X  was  the  tier  II  output  Promln  for  the 
targeted  year  at  the  respective  grade,  r  was  the  quantity  of  Marines  promoted  to 
E-2  for  targeted  year,  pulled  from  TFDW. 

Figure  37.  Promotions  to  E-2  Relative  Error 
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Points  were  calculated  with  Equation  6.  X  was  the  tier  II  output  Promln  for  the 
targeted  year  at  the  respective  grade,  r  was  the  quantity  of  Marines  promoted  to 
E-3  for  targeted  year,  pulled  from  TFDW. 

Figure  38.  Promotions  to  E-3  Relative  Error 
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Points  were  calculated  with  Equation  6.  X  was  the  tier  II  output  Promln  for  the 
targeted  year  at  the  respective  grade,  r  was  the  quantity  of  Marines  promoted  to 
E-4  for  targeted  year,  pulled  from  TFDW. 

Figure  39.  Promotions  to  E-4  Relative  Error 
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Points  were  calculated  with  Equation  6.  X  was  the  tier  II  output  Promln  for  the 
targeted  year  at  the  respective  grade,  r  was  the  quantity  of  Marines  promoted  to 
E-5  for  targeted  year,  pulled  from  TFDW. 

Figure  40.  Promotions  to  E-5  Relative  Error 
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Points  were  calculated  with  Equation  6.  X  was  the  tier  II  output  Promln  for  the 
targeted  year  at  the  respective  grade,  r  was  the  quantity  of  Marines  promoted  to 
E-6  for  targeted  year,  pulled  from  TFDW. 

Figure  41.  Promotions  to  E-6  Relative  Error 
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Points  were  calculated  with  Equation  6.  X  was  the  tier  II  output  Promln  for  the 
targeted  year  at  the  respective  grade,  r  was  the  quantity  of  Marines  promoted  to 
E-7  for  targeted  year,  pulled  from  TFDW. 

Figure  42.  Promotions  to  E-7  Relative  Error 
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Points  were  calculated  with  Equation  6.  X  was  the  tier  II  output  Promln  for  the 
targeted  year  at  the  respective  grade,  r  was  the  quantity  of  Marines  promoted  to 
E-8  for  targeted  year,  pulled  from  TFDW. 

Figure  43.  Promotions  to  E-8  Relative  Error 
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Points  were  calculated  with  Equation  6.  X  was  the  tier  II  output  Promln  for  the 
targeted  year  at  the  respective  grade,  r  was  the  quantity  of  Marines  promoted  to 
E-9  for  targeted  year,  pulled  from  TFDW. 

Figure  44.  Promotions  to  E-9  Relative  Error 
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Points  were  calculated  with  Equation  6.  X  was  output  End  for  the  targeted  year, 
r  was  the  total  quantity  of  Marines  within  TFDW  for  targeted  year. 

Figure  45.  End  Strength  Relative  Error 
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Points  were  calculated  with  Equation  6.  X  was  the  sum  of  the  outputs  A.i,  A.naf, 
and  A.o.  rwas  the  quantity  of  Marines  retained  within  Zone-A,  pulled  from 
TFDW  for  the  targeted  year. 

Figure  46.  Retention  Zone-A  Relative  Error 
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Points  were  calculated  with  Equation  6.  X  was  the  sum  of  the  outputs  B.i,  B.naf, 
and  B.o.  rwas  the  quantity  of  Marines  retained  within  Zone-B,  pulled  from 
TFDW  for  the  targeted  year. 

Figure  47.  Retention  Zone-B  Relative  Error 
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Points  were  calculated  with  Equation  6.  X  was  the  sum  of  the  outputs  C.i, 
C.naf,  and  C.o.  r  was  the  quantity  of  Marines  retained  within  Zone-C,  pulled  from 
TFDW  for  the  targeted  year. 

Figure  48.  Retention  Zone-C  Relative  Error 
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Points  were  calculated  with  Equation  6.  X  was  the  sum  of  the  outputs  D.i, 
D.naf,  and  D.o.  r  was  the  quantity  of  Marines  retained  within  Zone-D,  pulled  from 
TFDW  for  the  targeted  year. 

Figure  49.  Retention  Zone-D  Relative  Error 
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Points  were  calculated  with  Equation  6.  X  was  the  sum  of  the  outputs  E.i,  E.naf, 
and  E.o.  rwas  the  quantity  of  Marines  retained  within  Zone-E,  pulled  from 
TFDW  for  the  targeted  year. 

Figure  50.  Retention  Zone-E  Relative  Error 
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Points  were  calculated  with  Equation  6.  X  was  the  sum  of  the  outputs  SR.i, 
SR.naf,  and  SR.o.  r  was  the  quantity  of  Marines  retained  within  Zone-SR,  pulled 
from  TFDW  for  the  targeted  year. 

Figure  51 .  Retention  Zone-Senior-Enlisted  Relative  Error 
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D.  VALIDATION  ANALYSIS 

On  average,  the  simulation  runs  that  began  with  2009  and  2010  typically 
had  the  worst  relative  errors  for  each  category  This  consistency  through  the 
model  suggests  that  an  external,  uncontrolled  factor,  affected  manpower  data 
and  therefore  affected  the  MSM’s  validation.  Losses  and  Gains  showed  similar 
relative  error  characteristics.  Surprisingly,  both  these  graphs  produced  groupings 
of  minimal  error  that  clustered  in  the  second  to  forth  forecasted  year  range,  with 
the  larger  errors  occurring  immediately  in  simulation  and  then  toward  the  end. 
With  regard  to  promotion,  the  MSM  performed  well  when  forecasting  promotions 
to  the  Staff  Non-Commissioned  Officer  (SNCO)  ranks.  Promotions  to  Non- 
Commissioned  Officer  (NCO)  ranks  were  consistently  underestimated. 
Interestingly  though,  this  was  reciprocated  in  the  retention  zone  graphs  as  most 
of  the  graphs  consistently  overestimated  the  quantity  of  Marines  that  would 
reenlist.  End  Strength  overwhelmingly  returned  the  lowest  relative  error  out  of  all 
the  categories  evaluated.  This  once  again  reinforced  that  the  MSM’s  most 
important  input  is  the  GAR  inventory  structure  it  simulates  to. 

The  most  interesting  observation  was  the  correlation  between  reenlistment 
gains  and  accessions.  All  previous  investigation  identified  the  MSM  to  implement 
a  simple  supply  and  demand  philosophy;  once  a  billet  becomes  vacant  then  the 
MSM  will  exhaust  all  opportunities  to  move  a  Marine  Agent  into  the  spot  by  either 
a  lateral  move,  a  promotion,  or  a  lateral  move  and  a  promotion  simultaneously. 
What  was  interesting  to  discover  is  that  the  MSM  is  biased  in  choosing  Marine 
Agents  to  reenlist  vice  instantiating  Marine  Agents  through  the  recruit  and 
accessions  process.  This  is  seen  in  the  Results  Section  in  that  all  reenlistment 
zones  constantly  overestimated  targeted  values,  while  the  accessions  process 
continuously  underestimated  its  target  values. 
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VI.  CONCLUSION 


A.  EVALUATION 

This  thesis  had  three  main  objectives  to  accomplish.  The  first  was  to 
provide  a  document  that  explains  the  architecture  and  processes  of  the  MSM. 
The  second  was  to  examine  how  the  inputs  drive  the  model,  and  compare  to  the 
HRDP.  And  lastly,  we  had  the  objective  to  evaluate  the  accuracy  of  the  model  by 
a  validation  approach. 

The  MSM  is  a  large  complex  program.  It  totals  just  over  60  Java  classes 
for  its  source  code,  as  well  as  requires  eight  libraries  to  function.  This  program 
has  numerous  details  that  an  analysist  could  spend  days,  if  not  weeks,  diving  into 
and  investigating.  With  respect  to  the  objectives  of  this  thesis,  we  reached 
several  important  conclusions. 

•  The  MSM  is  a  simple  supply  and  demand  process. 

While  numerous  stochastic  processes  and  deterministic  algorithms  are 
included  within  the  MSM  to  forecast  manpower  metrics,  these  all  abide  by  the 
single  rule  that  a  billet  vacancy  must  exist  for  a  Marine  Agent  to  move  into  it. 
Likewise,  if  billet  vacancies  do  exist,  the  MSM  will  exhaust  every  inventory  option 
to  fill  any  empty  MOS-Grade  billets. 

•  The  GAR  input  is  the  most  important  input  that  drives  the  MSM  and 
its  manpower  forecasts. 

With  the  MSM  a  supply  and  demand  process,  the  most  important  input 
that  all  algorithms  are  run  against  is  the  GAR  structure.  The  GAR  sets  targeted 
billet  quantities  for  each  MOS  per  grade.  While  the  MSM  uses  stochastic 
processes  and  deterministic  algorithms  to  simulate  the  movement  of  Marine 
Agents  within  the  Marine  Corps,  the  MSM  will  always  try  to  fill  any  vacant  billet 
seat  in  order  to  achieve  an  inventory  that  maximizes  every  available  billet  seat 
specified  by  the  GAR.  Therefore,  the  MSM  could  be  viewed  more  as  a  tool  that 
evaluates  future  GAR  drafts. 
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•  The  MSM  offers  many  options  that  allow  a  user  to  fine  tune 
particular  parameters  for  a  simulation  run. 

At  its  foundation,  the  MSM  possesses  all  the  necessary  capabilities  to  run 
simulations  and  provide  forecasted  manpower  metrics.  Specifically,  it  uses 
historical  data  pulled  from  its  reference  inputs  to  set  the  likelihood  that  manpower 
events  occur,  and  guides  the  simulation  to  fully  populating  the  GAR  structure. 
Sitting  programmatically  on  top  of  that,  the  MSM  provides  numerous  fields  within 
its  “siminputs”  Excel  input  file  that  allows  the  user  to  fine  tune  simulation 
parameters.  It  is  recommended  to  only  use  these  tuning  parameters  when  small 
deviations  to  input  files  or  ad  hoc  case  scenarios  need  to  be  simulated. 
Examples  of  situations  range  from  adjusting  the  percentages  of  male  and  female 
Marines  per  allotted  end  strength  to  adjusting  overall  allowable  end  strength. 
Outside  of  ad  hoc  case  scenarios,  the  primary  effort  should  be  given  to  adjusting 
the  input  GAR  structure  and  selecting  archival  database  reference  input 
sequences  that  closely  align  with  the  manpower  goals  toward  which  M&RA  is 
looking  to  forecast. 

•  The  MSM  was  constructed  with  a  bottom-up  methodology  that 
captures  key  processes  within  the  HRDP. 

The  power  of  the  model  rests  in  its  ability  to  integrate  the  numerous 
processes  of  HRDP  into  a  single  simulation  model  and  simulate  down  into  the 
details  at  the  Marine  MOS-Grade  level.  Reinforcing  this  capability  is  the  plethora 
of  data  attributes  which  provide  the  user  with  output  measurements  that  can  be 
viewed  at  different  Tier  levels  depending  on  the  level  of  detail  the  user  wants  to 
drill  into.  The  Tier  I  level  provides  a  simple  aggregation  of  the  entire  Marine 
Corps  for  a  single  forecasted  year,  Tier  II  breaks  that  down  further  into  providing 
MOS-Grade  details  for  a  single  forecasted  year,  and  lastly  Tier  III  breaks  the  data 
down  even  further  to  provide  MOS-Grade  details  by  monthly  intervals. 

•  The  MSM  uses  a  modular  architecture  structure. 

Each  process  of  the  HRDP  was  constructed  as  its  own  Java  class,  and  is 
only  invoked  when  that  HRDP  is  required  to  be  executed.  This  greatly  increases 
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the  feasibility  of  editing  the  MSM  because  the  editor  just  needs  to  ensure  the 
parameter(s)  being  passed  to  and  returned  from  the  invoked  Java  class  remain 
aligned.  All  other  edits  can  be  made  within  a  process  with  little  dependencies  on 
the  remainder  of  the  MSM.  Besides  easiness  in  editing,  modularity  can  also 
facilitate  specific  policies  to  be  simulated  and  analyzed.  For  example,  if  M&RA 
wanted  to  update  policies  on  how  promotion  boards  are  run,  the  Promotions  Java 
class  could  easily  be  edited  to  reflect  a  drafted  policy,  which  would  ultimately 
produce  simulation  output  metrics  that  would  show  HRDP  whole  effects  to  a 
single  policy  change. 

•  The  MSM  is  a  time-step  model  with  the  base  time  increment  set  at 
one  fiscal  year. 

The  model  simulates  the  HRDP  by  employing  a  waterfall  technique  for  all 
the  processes,  where  a  bulk  ingest  queuing  methodology  is  utilized.  Here,  the 
entire  inventory  of  Marine  Agents  line  up  and  execute  an  itemized  process,  only 
to  move  onto  the  next  HRDP  process  after  the  last  Marine  Agent  finishes  the 
previous  one.  Additionally,  each  Marine  Agent  only  has  one  chance  at  executing 
a  certain  event  within  a  process,  and  then  that  Marine  Agent  will  have  to  wait  a 
full  fiscal  year  simulation  cycle  to  be  afforded  another  chance. 

•  MSM  employs  rudimentary  behavior  modeling  techniques  (artificial 
intelligence). 

For  Marine  Agents  navigating  their  careers  within  the  MOS-Grade  state 
graph  network,  basic  deterministic  rules  dictate  the  paths  that  Marine  Agents  will 
navigate.  They  are  simply  either  allowed  to  do  something  or  they  are  not;  a 
binary  decision.  The  only  behavior  modeling  capability  that  is  given  to  Marine 
Agents  is  in  the  form  of  statistical  probability  in  a  Markov  Chain  application.  For 
the  two  types  of  non-deterministic  processes  used,  a  Mersenne  Twister  random 
number  generator  picks  the  event  a  Marine  Agent  will  execute,  for  that  respective 
HRDP  process,  based  on  the  probabilities  of  the  respective  events. 

Overall,  the  MSM  is  an  exceptionally  powerful  tool  that  correctly  employs 
the  numerous  processes  of  the  HRDP.  While  the  level  of  accuracy  of  its  output 
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measures  varies  depending  on  which  output  measurement  is  being  examined, 
overall  it  provides  tangible  data  points  that  prior  to  this  model  could  have  only 
been  talked  about  in  theory.  For  that,  it  is  recommended  for  continued  use  and 
incorporating  into  plans  for  future  improvements. 

B.  RECOMMENDATIONS 

While  the  MSM  is  a  power  simulation  tool,  we  identified  various  areas 
were  continued  work  and  research  could  be  applied. 

•  Continued  Validation 

While  this  thesis  examined  the  MSM’s  input  sensitivity  with  respect  to 
eight  metamodels  and  validated  the  MSM  against  21  manpower  categories, 
plenty  of  opportunities  exist  to  dive  deeper  into  the  output  measures  and  validate 
their  results.  For  example  the  eight  metamodels  focused  primarily  at  the  whole 
Marine  Corps’  aggregated  level,  whereas  the  validation  also  drilled  down  into  the 
Grade  metric.  This  thesis  did  not  continue  examining  down  into  the  MOS-Grade 
levels,  Tiers  II  and  III  of  the  output  measures.  Additionally,  none  of  the  finance 
capabilities  of  the  MSM  were  examined  in  this  thesis. 

•  MSM  Parameter  Tuning 

The  validation  results,  illustrated  in  Chapter  V,  reveal  general  patterns  to 
MSM’s  output  metrics  accuracy  with  regard  to  which  year  in  the  future  it  is 
forecasting  for.  This  implies  that  parameters  could  be  weighted,  and  therefore 
tuned,  in  order  to  drive  that  output  metric  closer  to  a  zero  percentage  relative 
error. 

•  Intelligent  behavior 

As  previously  stated,  the  Marine  Agents  follow  simple  decision  rules  and 
the  simulated  selection  boards  are  likewise  simple.  There  are  a  wide  range  of 
aspects  and  areas  that  could  be  amended  with  additional  behavior 
characteristics  that  would  render  the  model  a  more  accurate  depiction  of  real-life 
Marine  Corps  manpower  decisions.  For  example,  a  reenlistment  bonus  behavior 
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could  be  constructed  that  improves  the  likelihood  of  a  Marine  Agent  to  reenlist 
based  upon  the  size  of  the  incentive. 

•  Add  geographical  code  and/or  Unit  Identification  Code  attributes  to 
Marine  Agents 

Just  as  the  Marine  Corps’  Table  of  Organization  allocates  billets  to  units, 
the  MSM  could  add  the  additional  Unit  Identification  Code  (UIC)  attribute  to  each 
Marine  Agent  in  order  to  track  additional  metrics.  Specifically,  the  geographical 
area  could  be  mapped  to  these  codes,  and  therefore  a  Permanent  Change  of 
Station  Java  class  could  be  constructed  that  forecasts  the  financial  surge 
associated  with  Marines  moving  to  new  commands  in  new  geographical  areas. 
Likewise,  a  UIC  attribute  could  be  added  to  each  Marine  Agent.  This  attribute 
could  be  used  to  simulate  staffing  goals  for  units  as  well  as  more  accurately  map 
the  career  progression  of  a  Marine  Agent  traversing  the  MOS-Grade  state  graph 
network. 

•  Introduce  a  Graphic  User  Interface  (GUI) 

Numerous  steps  exist  for  setting  up  and  running  the  MSM.  From  loading 
all  the  inputs  to  setting  user  defined  parameters  within  the  “siminputs”  input  file, 
these  steps  require  an  intricate  level  of  knowledge  about  the  MSM  in  order  to  be 
done  correctly.  Converting  the  user  interaction  with  the  MSM  into  a  GUI  would 
better  facilitate  the  process  for  the  user  to  accurately  set  up  and  run  the  model. 

•  MSM  output  data  type 

The  current  version  of  the  MSM  writes  its  data  outputs  to  a  text-type 
document.  A  few  modifications  to  the  outputs  Java  class  could  transcribe  these 
documents  into  Excel  files,  such  as  comma  separated  files.  Additionally, 
modifications  to  the  Java  class  could  automatically  produce  specifically  sought 
after  visual  graphics. 

•  Remove  MSM’s  retentions  over  accessions  bias 

A  negative  aspect  of  the  MSM  is  that  it  overestimates  Marine  Agents  that 
choose  to  reenlist.  This  has  consequences  that  trickle  down  through  the  MOS- 

Grade  state  graph  network  and  directly  affects  the  accessions  forecasts. 
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Specifically,  because  the  MSM  employs  a  supply  and  demand  methodology  in 
exhausting  all  opportunities  to  retain  a  Marine  Agent,  any  Marine  Agents  retained 
are  ultimately  occupying  billets  that  make  it  more  difficult  for  lower  grade  Marine 
Agents  to  promote  upwards.  This  impacts  the  inventory  for  each  MOS-Grade, 
which  ultimately  restricts  the  quantity  of  accessions  the  MSM  can  instantiate  to 
backfill  billet  vacancies  that  have  trickled  their  way  down  to  the  entry  level  MOS- 
Grades. 
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APPENDIX  A.  MSM  MODEL  INPUTS 


Table  5.  Main  Sheet  Inputs 


Parameter 

Description 

TFDW  User  Name 

User  name  if  pulling  from  online  database 

TFDW  Password 

Password  if  pulling  from  online  database 

ODSE  User  Name 

User  name  if  pulling  from  online  database 

ODSE  Password 

Password  if  pulling  from  online  database 

Scenario  Name 

Optional  scenario  name  for  simulation  run 

Number  of  Years 

Number  of  years  to  forecast 

Number  of 

Simulations 

Number  of  repetitions  to  run  the  stochastic  simulation 

Inventory 

Sequence  number  from  which  to  pull  inventory  data 

Initial  Forecasted 

Year 

The  first  year  that  will  be  forecasted 

Losses 

If  blank  program  pulls  local  file. 

Enter  comma  separated  list  to  pull  those  years  from  TFDW/ODSE. 

Reenlistments 

If  blank  program  pulls  local  file. 

Enter  comma  separated  list  to  pull  those  years  from  TFDW/ODSE. 

Gains 

If  blank  program  pulls  local  file. 

Enter  comma  separated  list  to  pull  those  years  from  TFDW/ODSE. 

Promotions 

If  blank  program  pulls  local  file. 

Enter  comma  separated  list  to  pull  those  years  from  TFDW/ODSE. 

Report  Options 

Establishes  how  the  program  will  aggregate  its  results. 

-  Year 

-  Year  and  MOS 

-  Year  and  Grade 

-  Year,  MOS,  and  Grade 

-  Year  and  Month 

-  Year,  Month,  and  Grade 

-  Year,  Month,  MOS,  and  Grade 

Table  6.  Career  Force  Controls 


Parameter 

Description 

Grade 

Grade  (E1-E9).  Each  subsequent  parameter  is  broken  down  and 
evaluated  per  grade. 

Promotion  Targets 

Promotion  targets  for  subject  grade 

Minimum  TIS 

Controls  eligibility  for  promotion 

Maximum  TIS 

Controls  eligibility  for  promotion  and  retention 

Minimum  TIG 

Controls  eligibility  from  promotion 
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Table  6  Continued.  Career  Force  Controls 


Parameter 

Description 

Variable  In-Zone 
Selection 

Opportunity 

Controls  zone  size,  selection  rate,  and  passover  rate. 

Maximum  # 
of  Passovers 

Controls  whether  Marine  may  continue  service 

Limit  Type  if  exceed 
maximum  passovers 

Indicates  type  of  control  to  place  if  passover 
limit  is  reached 

TIS  Limit  if  exceed 
maximum  passovers 

TIS  limit  adjustment  for  those 
who've  been  passed  over 

Table  7.  MOS  Data 


Parameter 

Description 

PMOS 

Digit  indicator  of  primary  MOS 

Gender 

Gender  indicator,  used  on  MOS  restrictions 

Is  BMOS 

Indicates  a  basic/school  training  MOS 

Time  to  Train 
(Non  LM) 

Number  of  days  spent  in  training 
pipeline 

Time  to  Train 
(LM) 

Number  of  days  spent  in  training 
when  lat  moving  to  this  MOS 

Lat  Move  Only 

Indicates  if  this  MOS  is  a  'Lat  Move  Only’  MOS 

Title 

Name  of  MOS 

Converting  MOSs 

List  of  MOSs  that  are  obsolete  predecessors  of  PMOS 

Alias 

List  of  MOSs  that  PMOS  has  been  known  as  in  the  past 

Comparable  MOSs 

List  of  MOSs  that  can  be  used  in  place  of  subject  MOS 

Feeds  From 

List  of  MOSs  that  feed  into  the  subject  MOS 

Feeds  To 

List  of  MOSs  that  the  subject  MOS  feeds  into 

Min  Grade 

Minimum  grade  for  subject  MOS 

Max  Grade 

Maximum  grade  for  subject  MOS 

TOE  (4) 

4-Year  contract 

TOE  (5) 

5-Year  contract 

TOE  (6) 

6-Year  contract 
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APPENDIX  B.  GAR  INPUT 


The  GAR  is  a  user-provided  input  to  the  MSM  program.  Its  data  is 
responsible  for  building  the  MOS-Grade  state  graph  network  of  the  model,  which 
serves  as  the  manpower  structure  that  the  Marine  Agents  navigate  through 
during  the  simulation  of  their  career.  The  below  image  is  a  snapshot  of  this  input: 


TARGET  FORCE  PLANNING  MODEL  ENLISTED  GAR  BUILD  REPORTS 
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0 
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0 
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Figure  52.  Snapshot  of  GAR  Input. 
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APPENDIX  C.  OUTPUT  METRICS 


Table  8.  MSM  Output  Metrics 


Category 

Metric 

Definition 

A-Bil 

Pulled  from  the  GAR  input,  these  are  primary,  direct  billets  without  allocation  of  overhead,  b-billets,  training,  etc.  The  "naive"  requirement. 

CD 

B-Bil 

Pulled  from  the  GAR  input,  these  are  B-Billets  allocated  to  structure. 

o 

g 

P2T2.gar 

Pulled  from  the  GAR  input,  these  are  "Overhead"  allocated  for  patients,  prisoners,  trainees,  and  transients. 

"5 

Q. 

c 

Gar.u 

This  is  unadjusted  "raw"  GAR  requirement  which  doesn't  account  for  allocation  of  those  in  boot  camp  or  MOS  school.  Its  formula  is: 

in 

0 

Gar.u  =  A-Bil  +  B-Bil  +  P2T2.gar 

0 

.Q 

Train. Boot 

Proportional  allocation  to  account  for  Marines  in  Boot  Camp. 

< 

s 

0 

Q 

Train. Mos 

Proportional  allocation  to  account  for  Marine  in  MOS  School 

GAR  adjusted  for  Boot  Camp  and  MOS  School  allocation.  This  is  the  calculation  used  to  build  the  MOS-Grade  state  graph  network.  Its 

O 

Gar. a 

formula  is: 

CO 

Gar.a  =  Gar.u  +  Train. Boot  +  Train. Mos 

E 

E 

Start 

Starting  inventory  pulled  from  TFDW  data.  Always  based  on  FY  Start  (1  Oct) 

C/) 

FemStart 

Starting  female  inventory 

Neas.n 

Quantity  of  Marine  Agents  selected  to  exit  the  Marine  Corps  with  a  Non-EAS  separation  code. 

to  tO 

0  < 

O  LU  0 

0  i  ~0 

£  !  o 
>,  =  g 

Deaths. n 

Quantity  of  Marine  Agents  selected  to  exit  the  Marine  Corps  due  to  death. 

E20.n 

Quantity  of  Marine  Agents  selected  to  receive  an  enlisted  to  officer  (e2o)  commission,  and  therefore  leave  the  enlisted  manpower  structure. 

o-B- 
O  '>  0 

0  > 

Retired. n 

Quantity  of  Marine  Agents  selected  to  exit  the  Marine  Corps  due  to  retiring. 

0  -O  Q. 

O  .5?  0 

o  o 
°  O 

Other.n 

Quantity  of  Marine  Agents  selected  to  exit  the  Marine  Corps  due  to  other  means  not  documented  with  a  specific  separation  code. 

Eas.n 

Quantity  of  Marine  Agents  selected  to  exit  the  Marine  Corps  due  to  the  end  of  their  enlisted  contract  called  End  of  Active  Service  (EAS). 

Boot.n 

Quantity  of  Marine  Agents  selected  to  exit  the  Marine  Corps  during  bootcamp. 
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Table  8  Continued.  MSM  Output  Metrics 


Category 

Metric 

Definition 

w 

Neas.e 

Quantity  of  Marine  Agents  selected  to  exit  the  Marine  Corps  with  a  Non-EAS  separation  code. 

GO  _ 

o  co  0 

CD  <  -o 

Deaths. e 

Quantity  of  Marine  Agents  selected  to  exit  the  Marine  Corps  due  to  death. 

E20.e 

Quantity  of  Marine  Agents  selected  to  receive  an  enlisted  to  officer  (e2o)  commission,  and  therefore  leave  the  enlisted  manpower  structure. 

if  i 

Retired. e 

Quantity  of  Marine  Agents  selected  to  exit  the  Marine  Corps  due  to  retiring. 

f  S  « 

CO  Q. 

O  "O  0 

«  °  W 

C/5  CJ 

o 

Other.e 

Quantity  of  Marine  Agents  selected  to  exit  the  Marine  Corps  due  to  other  means  not  documented  with  a  specific  separation  code. 

Eas.e 

Quantity  of  Marine  Agents  selected  to  exit  the  Marine  Corps  due  to  the  end  of  their  enlisted  contract  called  End  of  Active  Service  (EAS). 

_l 

Boot.e 

Quantity  of  Marine  Agents  selected  to  exit  the  Marine  Corps  during  bootcamp. 

0 

Ead.g 

Quantity  of  Marine  Agents  selected  that  return  from  their  extended  active  duty  (EAD)  recruiting  tour. 

CO 

CO 

O 

05 

O 

LL 

Occ.g 

Quantity  of  Marine  Agents  selected  that  return  after  having  become  a  deserter. 

Rev.g 

Quantity  of  Marine  Agents  selected  that  return  to  service  with  a  Rever  gain  code. 

O 

05 

Cont.g 

Quantity  of  Marine  Agents  selected  that  return  to  service  with  a  Cont  gain  code. 

05 

CO 

O 

C 

'cO 

Brok.g 

Quantity  of  Marine  Agents  selected  that  return  from  medical  holds. 

Des.g 

Quantity  of  Marine  Agents  selected  that  return  after  having  become  a  deserter. 

0 

Other.g 

Quantity  of  Marine  Agents  selected  that  return  to  service  for  other  means  not  captured. 

Accession 

Forecasts 

NPS.g 

Non-prior  service  accessions.  This  field  is  calculated  based  on  vacancies  required  to  fill  requirements  or  is  supplied  by  the  user. 

NPSFem.g 

Same  as  NPS.g,  but  for  females.  There  is  no  required  female  recruiting,  so  this  is  an  estimate  based  on  data. 

Promln 

Quantity  of  Marine  Agents  promoted  into  category  (depending  on  tier  level,  could  be  aggregrate  FY,  by  MOS-Grade,  by  month. ..ect). 

C/5 

O 

PromOut 

Quantity  of  Marine  Agents  promoted  out  of  a  categoy  (depending  on  tier  level,  could  be  aggregrate  FY,  by  MOS-Grade,  by  month. ..ect). 

05 

LMIn 

Quantity  of  Marine  Agents  lateral  moved  into  category  (depending  on  tier  level,  could  be  aggregrate  FY,  by  MOS-Grade,  by  month. ..ect). 

© 

£ 

LMOut 

Quantity  of  Marine  Agents  lateral  moved  out  of  a  categoy  (depending  on  tier  level,  could  be  aggregrate  FY,  by  MOS-Grade,  by  month. ..ect). 

o 

Q_ 

c 

CO 

End 

Quantity  of  Marine  Agents  in  the  inventory  at  the  end  of  the  time  iteration  (could  be  either  by  FY  or  by  month) 

FemEnd 

Quantity  of  female  Marine  Agents  in  the  inventory  at  the  end  of  the  time  iteration  (could  be  either  by  FY  or  by  month) 

"o 

© 

c 

P2T2 

Quantity  of  Marine  Agents  for  the  aggregated  time  period  (could  be  either  by  FY  or  by  month,  depends  on  which  output  tier  viewed). 

05 

0 

BasePay 

Estimated  Base  Pay  for  inventory  based  on  pay  tables 

CompPay 

Estimated  Composite  pay  based  on  by-grade  composite  planning  factors. 
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Table  8  Continued.  MSM  Output  Metrics 


Category 

Metric 

Definition 

EligA 

Quantity  of  Marine  Agents  eligible  for  retention  in  zone  A. 

EligB 

Quantity  of  Marine  Agents  eligible  for  retention  in  zone  B. 

EligC 

Quantity  of  Marine  Agents  eligible  for  retention  in  zone  C. 

EligD 

Quantity  of  Marine  Agents  eligible  for  retention  in  zone  D. 

EligE 

Quantity  of  Marine  Agents  eligible  for  retention  in  zone  E. 

EligSR 

Quantity  of  Marine  Agents  eligible  for  retention  in  zone  SR. 

A.i 

Quantity  of  Zone  A  Marine  Agents  who  reenlisted  "In-Year.” 

B.i 

Quantity  of  Zone  B  Marine  Agents  who  reenlisted  "In-Year.” 

C.i 

Quantity  of  Zone  C  Marine  Agents  who  reenlisted  "In-Year.” 

(/) 

o 

D.i 

Quantity  of  Zone  D  Marine  Agents  who  reenlisted  "In-Year.” 

0 

E.i 

Quantity  of  Zone  E  Marine  Agents  who  reenlisted  "In-Year.” 

c 

o 

SR.i 

Quantity  of  Zone  SR  Marine  Agents  who  reenlisted  "In-Year.” 

0 

0 

A.naf 

Marine  Agents  that  reenlisted  into  zone  A  that  were  "Not  Accounted  For"  (see  end  note). 

cr 

B.naf 

Marine  Agents  that  reenlisted  into  zone  B  that  were  "Not  Accounted  For"  (see  end  note). 

C.naf 

Marine  Agents  that  reenlisted  into  zone  C  that  were  "Not  Accounted  For"  (see  end  note). 

D.naf 

Marine  Agents  that  reenlisted  into  zone  D  that  were  "Not  Accounted  For"  (see  end  note). 

E.naf 

Marine  Agents  that  reenlisted  into  zone  E  that  were  "Not  Accounted  For"  (see  end  note). 

SR.naf 

Marine  Agents  that  reenlisted  into  zone  SR  that  were  "Not  Accounted  For”  (see  end  note). 

A.o 

Quantity  of  Zone  A  Marine  Agents  who  reenlisted  "Out-of-Year”  (i.e.  EAS  in  2017  but  Marine  Agent  reenlisted  in  2016). 

B.o 

Quantity  of  Zone  B  Marine  Agents  who  reenlisted  "Out-of-Year"  (i.e.  EAS  in  2017  but  Marine  Agent  reenlisted  in  2016). 

C.o 

Quantity  of  Zone  C  Marine  Agents  who  reenlisted  "Out-of-Year"  (i.e.  EAS  in  2017  but  Marine  Agent  reenlisted  in  2016). 

D.o 

Quantity  of  Zone  D  Marine  Agents  who  reenlisted  "Out-of-Year"  (i.e.  EAS  in  2017  but  Marine  Agent  reenlisted  in  2016). 

E.o 

Quantity  of  Zone  E  Marine  Agents  who  reenlisted  "Out-of-Year”  (i.e.  EAS  in  2017  but  Marine  Agent  reenlisted  in  2016). 

SR.o 

Quantity  of  Zone  SR  Marine  Agents  who  reenlisted  "Out-of-Year"  (i.e.  EAS  in  2017  but  Marine  Agent  reenlisted  in  2016). 
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Table  8  Continued.  MSM  Output  Metrics 


Category 

Metric 

Definition 

Other  Manpower 
Metrics 

Passovers 

Number  of  times  Marine  Agents  are  passed  over  for  promotion,  aggregated. 

PromZoneElig 

Number  of  Marine  Agents  eligible  for  promotion  and  placed  in  zone,  aggregated. 

PromElig 

Number  of  Marine  Agents  eligible  for  promotion,  irrespective  of  whether  or  not  they  were  placed  in  zone. 

sumTTP 

The  aggregated  sum  of  Time  to  Promotion. 

sumTIG 

The  aggregated  sum  of  Time  in  Grade. 

sumYOS 

The  aggregated  sum  of  Years  of  Service. 

The  ,”naf”  category  is  an  artifact  of  archival  bad  data.  Prior  to  the  construction  of  the  MSM,  it  was  identified  by  M&RA  that  there 
was  a  leakage  within  the  retention  accountability  system  due  to  various  HRDP  processes  and  how  they  interfaced  together. 
Specifically,  there  were  Marines  who  were  reenlisting  that  were  not  being  accounted  for  by  either  the  “in-year”  or  “out-of-year” 
retention  categories.  To  account  for  this  group  of  Marines  who  were  actually  reenlisting  but  not  being  properly  accounted  for,  the 
developer  built  the  “not-accounted-for”  retention  category  within  the  MSM  (R.  A.  Garrick,  personal  communication,  April  6,  2016). 
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APPENDIX  D.  DESIGN  ORTHOGONALITY 


Table  9.  Design  Orthogonality  of  NOLH  DOE 
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-0.003 

-0.013 

0004 

0007 

0.008 

-3E-04 

0.001 

1 

0.007 

-0.01 

-0  007 

GCont 

0.007 

0.003 

0.01 

0.003 

0.003 

0.003 

0.005 

-0  009 

-0.06 

-0.007 

-0.003 

-4E-04 

0.007 

0.005 

0.005 

0.002 

0  006 

0.005 

0.003 

-2E-04 

0.007 

1 

0.005 

0.005 

GRever 

-0.011 

0.011 

0031 

-0.006 

-0.005 

-0.003 

-0.008 

0011 

0.012 

0.01 

0.003 

6E-04 

-0.009 

-0.005 

-0.011 

-0  003 

-0  008 

001 

-7E-04 

0.001 

-0.01 

0.005 

1 

-0.008 

G.Brkn 

-0  029 

0.007 

0.02 

-0.006 

-0.005 

-0.002 

-0.008 

-0.049 

-0.017 

0.007 

0.004 

3E-05 

-0.008 

-0.003 

-0.007 

-0.003 

-0.004 

0.005 

-0  002 

4E-04 

-0  007 

0  005 

-0.008 

1 

Within  the  table,  the  top  three  correlations  (the  three  worst  orthogonalities)  are  highlighted  in  yellow. 
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Table  10.  NOLH  Design 


low  level 

-2 

-* 

-1 

-2 

-0.1 

-0.05 

-0.05 

-2 

-2 

0.6 

0.6 

0.6 

0.6 

0.6 

0.6 

0.6 

0.6 

0.6 

0.6 

0.6 

0.6 

0.6 

0.6 

0.6 

high  level 

3 

4 

3 

4 

0.05 

0.05 

0.1 

2 

2 

1.1 

1.1 

1.1 

1.1 

1.1 

1.1 

1.1 

1.1 

1.1 

1.1 

1.1 

1.1 

1.1 

1.1 

1.1 

decimals 

0 

0 

0 

1 

3 

3 

3 

0 

0 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

TIS 

TISmax 

TIG 

Prom 

Pass 

Reinlist 

Loss 

Loss 

Loss 

Loss 

Loss 

Loss 

Loss 

Loss 

Gain 

Gain 

Gain 

Gain 

Gain 

Gain 

Gain 

factor  name 

min 

PassO  max 

min 

Tgt 

VIZmin 

VizStd 

VIZmax 

over 

Thresh 

Boot 

Retired 

NEAS 

Death 

e2o 

Other 

EAS 

Reup 

EAD 

Desert  Other 

OCC 

Cont 

Rever 

Broken 

0 

2 

-1 

-0.6 

0.015 

-0.028 

0.023 

2 

1 

0.713 

1.055 

1.043 

0.704 

0.659 

0.914 

0.621 

0.637 

0.926 

1.073 

0.696 

0.709 

1.004 

0.877 

1.065 

-1 

-1 

0 

-1.6 

0.005 

-0.032 

-0.039 

1 

-2 

0.963 

0.772 

0.821 

1.1 

0.772 

0.877 

0.625 

0.666 

0.952 

1.09 

0.666 

0.838 

0.83 

0.844 

0.938 

0 

2 

2 

4 

-0.061 

-0.004 

0.008 

1 

-1 

0.905 

0.877 

1.065 

0.676 

0.668 

0.961 

1.055 

0.913 

0.627 

1.088 

0.881 

0.895 

0.647 

0.776 

0.709 

-1 

-2 

2 

3.8 

-0.035 

-0.018 

-0.044 

1 

1 

0.768 

0.799 

0.789 

0.957 

0.633 

1.098 

0.903 

0.866 

0.668 

0.959 

0.868 

1.073 

0.786 

0.743 

0.834 

-1 

2 

0 

3.6 

-0.096 

-0.011 

0.016 

1 

1 

1 

0.93 

0.887 

0.641 

1.098 

0.819 

0.862 

0.631 

1.01 

1.096 

1.075 

1.018 

0.678 

0.729 

0.655 

-1 

-2 

0 

3.3 

-0.065 

-0.02 

0.005 

2 

-1 

0.62 

0.76 

0.801 

0.998 

1.1 

0.75 

0.864 

0.793 

0.948 

0.916 

0.95 

0.889 

0.674 

0.633 

0.62 

-1 

4 

3 

-0.5 

-0.047 

-0.034 

0.011 

1 

-2 

1.096 

0.954 

1.012 

0.664 

0.971 

1.041 

0.846 

0.918 

0.659 

0.651 

0.76 

0.823 

1.098 

0.705 

0.776 

-2 

-3 

3 

-0.4 

-0.098 

-0.025 

-0.026 

1 

2 

0.77 

0.651 

0.793 

0.897 

1.036 

0.922 

0.973 

1.073 

0.809 

0.6 

0.778 

0.864 

1.034 

0.766 

0.604 

-2 

4 

-1 

-0.6 

-0.064 

-0.017 

-0.032 

1 

2 

1.055 

0.938 

0.875 

0.637 

0.731 

0.604 

0.745 

0.686 

1.071 

0.811 

0.879 

0.614 

1.002 

0.618 

0.668 

-2 

-4 

-1 

-0.2 

-0.091 

-0.004 

0.01 

2 

-2 

0.69 

0.786 

0.643 

0.907 

0.803 

0.77 

0.723 

0.83 

0.963 

0.801 

0.756 

0.788 

0.936 

0.795 

0.825 

-2 

1 

2 

2.5 

-0.021 

0.021 

0.081 

-2 

-1 

0.668 

0.766 

0.69 

1.096 

0.618 

0.827 

0.635 

0.602 

1.03 

0.657 

0.649 

0.782 

0.924 

0.774 

0.684 

0 

-3 

2 

2.9 

0.038 

0.008 

0.077 

-2 

1 

0.887 

0.92 

1.053 

0.606 

0.645 

0.711 

0.684 

0.606 

1.012 

0.637 

0.766 

0.795 

0.995 

0.784 

0.842 

-1 

2 

0 

3 

0.029 

0.007 

0.084 

-2 

1 

0.799 

0.606 

0.641 

1.069 

0.918 

1.09 

0.934 

0.989 

0.614 

0.795 

0.834 

0.655 

0.979 

0.647 

0.799 

-1 

-2 

-1 

3.4 

-0.003 

0.047 

0.092 

-2 

-1 

1.045 

0.87 

1.071 

0.67 

0.938 

0.936 

0.844 

1.051 

0.6 

0.901 

0.782 

0.717 

1 

0.821 

0.786 

-1 

3 

2 

-0.2 

-0.017 

0.033 

0.053 

-1 

-1 

0.774 

0.807 

0.653 

0.965 

1.026 

0.715 

0.731 

0.731 

0.965 

0.895 

1.065 

1.075 

0.649 

0.748 

0.778 

-1 

-3 

2 

-0.8 

-0.002 

0.008 

0.093 

-1 

1 

0.86 

1.002 

0.961 

0.727 

0.944 

0.606 

0.637 

0.825 

0.938 

1.028 

0.911 

1.028 

0.629 

0.639 

0.791 

-1 

2 

0 

0.2 

0.049 

0.019 

0.038 

-1 

2 

0.663 

0.789 

0.784 

0.983 

0.627 

0.957 

0.852 

1.047 

0.741 

1.061 

0.913 

1.014 

0.6 

0.731 

0.764 

-1 

-3 

-1 

-1.5 

0.027 

0.021 

0.052 

-1 

-1 

1.075 

1.014 

0.901 

0.748 

0.723 

0.862 

0.932 

0.987 

0.733 

0.928 

0.975 

0.998 

0.637 

0.702 

0.621 

-1 

4 

1 

1.1 

-0.064 

0.021 

0.097 

-1 

-1 

1.014 

0.752 

0.83 

0.881 

0.735 

1.059 

0.83 

0.809 

0.987 

1.082 

0.864 

0.694 

1.088 

0.897 

0.961 

-2 

-3 

2 

3.5 

-0.073 

0.012 

0.033 

0 

0 

0.745 

0.905 

1.092 

0.721 

0.707 

1.047 

0.608 

0.784 

0.995 

0.92 

0.799 

0.807 

0.955 

0.93 

0.926 

0 

0 

-1 

3.2 

-0.074 

0.045 

0.04 

-2 

1 

1.071 

0.797 

0.811 

1 

1.018 

0.733 

0.93 

0.946 

0.791 

1.047 

0.762 

0.844 

1.096 

0.862 

1.038 

0 

-1 

0 

1.9 

-0.084 

0.013 

0.043 

-1 

-2 

0.696 

1.084 

0.893 

0.61 

0.926 

0.791 

0.916 

1.075 

0.661 

1.041 

0.87 

0.696 

0.901 

0.989 

0.985 

0 

3 

1 

0.7 

-0.055 

0.001 

0.05 

-1 

-2 

1.012 

0.631 

0.836 

1.086 

0.979 

1.065 

0.829 

0.694 

1.077 

0.641 

0.875 

0.969 

0.729 

1.051 

1.077 

-1 

-1 

3 

0.1 

-0.071 

0.04 

0.027 

-2 

0 

0.748 

0.844 

0.965 

0.758 

1.086 

0.92 

0.772 

0.813 

0.928 

0.678 

1.079 

0.891 

0.606 

0.899 

1.043 

-2 

0 

0 

-2 

-0.046 

0.009 

0.034 

-1 

1 

1.028 

0.873 

0.666 

1.084 

0.752 

0.696 

1.028 

0.92 

0.653 

0.682 

0.979 

0.901 

0.856 

0.809 

0.954 

0 

-3 

0 

-1.1 

-0.087 

0.005 

0.078 

-2 

-1 

0.782 

0.916 

0.873 

0.715 

0.87 

0.729 

0.834 

0.936 

0.743 

0.803 

0.971 

1.069 

0.711 

0.944 

1.075 

0 

1 

1 

2.3 

0.006 

-0.002 

C.C-9 

-1 

-2 

0.827 

1.059 

0.768 

1.034 

0.815 

0.621 

0.95 

0.651 

1.016 

0.729 

0.973 

0.758 

1.036 

1.03 

0.645 
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0 

0 

3 

1.7 

-0.013 

-0.042 

0.051 

-1 

0 

0.989 

0.809 

0.92 

0.75 

0.612 

0.83 

0.987 

0.647 

1.059 

0.762 

1.084 

0.776 

1.038 

0.871 

0.639 

2 

0 

1 

3.7 

-0.014 

-0.036 

0.068 

-1 

0 

0.661 

0.895 

0.62 

0.993 

0.932 

0.926 

0.663 

0.873 

0.649 

0.807 

0.893 

0.715 

1.018 

1.006 

0.803 

0 

-3 

1 

1.8 

0.045 

-0.015 

0.088 

-1 

-1 

0.897 

0.657 

1.002 

0.682 

0.899 

0.998 

0.707 

0.854 

0.704 

0.754 

0.873 

0.637 

1.084 

0.911 

0.717 

0 

4 

2 

1 

0.025 

-0.005 

0.095 

-2 

-1 

0.823 

0.909 

0.694 

1.038 

0.832 

0.807 

1.004 

0.729 

1.049 

1.069 

0.752 

1.071 

0.735 

0.975 

0.817 

2 

-1 

1 

-1.4 

-0.013 

-0.015 

0.074 

-1 

1 

1.008 

0.838 

1.084 

0.602 

0.879 

0.707 

1.012 

0.704 

0.993 

0.903 

0.741 

0.993 

0.676 

1.018 

0.68 

0 

2 

-1 

0.5 

0.004 

-0.049 

0.059 

-1 

0 

0.754 

1.071 

0.748 

1.02 

0.678 

0.871 

0.786 

1.002 

0.68 

0.844 

0.651 

0.932 

0.657 

0.95 

0.618 

1 

1 

0.7 

0.007 

-0.035 

0.048 

-1 

-2 

0.967 

0.62 

0.942 

0.705 

0.807 

1.006 

0.799 

1.018 

0.612 

1.03 

0.829 

0.975 

0.766 

0.979 

0.643 

1 

2 

2 

2.2 

-0.057 

-0.026 

0.099 

0 

-1 

0.914 

0.95 

0.649 

0.92 

0.629 

0.959 

1.077 

0.856 

0.979 

0.86 

1.055 

0.649 

0.94 

0.643 

0.932 

0 

-2 

1 

1.5 

-0.043 

-0.032 

0.083 

-1 

1 

0.76 

0.68 

1.086 

0.823 

0.836 

1.069 

0.938 

0.633 

0.842 

0.911 

0.93 

0.663 

0.895 

0.604 

0.934 

0 

3 

0 

1.4 

-0.092 

-0.033 

0.071 

0 

1 

0.909 

1.039 

0.803 

0.934 

0.955 

0.776 

0.764 

1.08 

0.764 

0.983 

1.094 

0.83 

0.983 

0.791 

0.973 

1 

-1 

0 

3.8 

-0.044 

-0.039 

0.031 

-1 

-1 

0.676 

0.666 

0.981 

0.795 

0.975 

0.735 

0.719 

1.022 

0.788 

0.862 

0.957 

0.735 

0.952 

0.834 

1.09 

2 

1 

2 

-1 

-0.026 

-0.02 

0.066 

-1 

0 

1.051 

1.004 

0.815 

0.959 

0.961 

0.86 

1.03 

0.664 

0.901 

0.77 

0.631 

0.887 

0.813 

0.739 

1.047 

0 

-4 

1 

0.5 

-0.085 

-0.03 

0.036 

-1 

1 

0.856 

0.731 

0.963 

0.659 

0.998 

0.854 

0.885 

0.61 

0.905 

0.663 

0.678 

1.043 

0.768 

0.696 

1.012 

0 

2 

1 

-0.1 

-0.038 

-0.014 

0.087 

-1 

2 

0.864 

1.065 

0.633 

0.87 

0.84 

0.682 

0.659 

0.905 

0.746 

0.735 

0.784 

1.055 

0.788 

0.606 

0.864 

1 

-1 

-1 

-0.3 

-0.032 

-0.041 

0.09 

0 

0 

0.797 

0.821 

1.038 

0.772 

0.719 

0.799 

0.704 

0.973 

0.643 

0.766 

0.801 

1.096 

0.789 

0.811 

0.928 

2 

3 

2 

4 

-0.001 

0.038 

-0.006 

-2 

0 

0.807 

0.647 

0.909 

0.875 

0.737 

0.637 

0.666 

1.041 

0.95 

0.733 

0.776 

0.971 

0.957 

0.664 

0.891 

2 

-3 

1 

3.3 

0.001 

0.034 

0.013 

0 

1 

0.938 

1.026 

0.774 

0.668 

0.69 

0.815 

0.614 

0.883 

0.971 

0.75 

0.817 

0.922 

0.905 

0.817 

0.893 

0 

2 

-1 

2.9 

0.018 

0.039 

0.014 

-1 

2 

0.651 

0.649 

1.098 

1.01 

0.985 

0.911 

1.051 

0.758 

0.768 

0.684 

0.668 

0.866 

0.985 

0.836 

1 

1 

-1 

-1 

1.3 

0.038 

0.045 

-0.045 

-1 

-2 

1.006 

1.1 

0.844 

0.739 

1.075 

0.975 

1.018 

0.762 

0.657 

0.68 

0.709 

0.934 

1.057 

0.745 

0.854 

0 

2 

3 

-0.6 

0.045 

0.019 

-0.025 

-1 

-i 

0.682 

0.637 

0.922 

1.053 

0.983 

0.731 

0.741 

0.909 

1.008 

1.01 

1.002 

0.618 

0.686 

0.793 

0.916 

1 

0 

3 

0.6 

0.024 

0.045 

0.013 

-2 

2 

1.088 

0.952 

0.723 

0.719 

0.995 

0.788 

0.805 

0.981 

0.887 

0.975 

1.09 

0.791 

0.772 

0.848 

1.051 

1 

4 

0 

-1.5 

0.009 

0.009 

-0.004 

0 

0 

0.743 

0.832 

0.969 

0.991 

0.795 

0.977 

1.061 

0.803 

0.663 

0.918 

1.063 

0.797 

0.801 

0.764 

0.852 

2 

-3 

1 

-1.6 

-0.002 

0.018 

-0.007 

-1 

-1 

0.948 

0.955 

0.766 

0.848 

0.686 

1.075 

0.92 

0.709 

0.745 

0.995 

0.954 

0.733 

0.672 

0.602 

0.989 

0 

2 

3 

2 

-0.099 

0.048 

-0.007 

-1 

-1 

1.067 

0.782 

0.93 

0.879 

0.606 

0.875 

0.825 

0.961 

0.924 

1.067 

0.657 

0.897 

1.006 

0.901 

0.805 

1 

-1 

2 

2.1 

-0.083 

0.005 

0.007 

0 

2 

0.61 

1.057 

0.75 

0.688 

0.793 

1.073 

0.819 

1.059 

0.83 

1.051 

0.813 

0.858 

0.866 

0.903 

0.743 

1 

4 

0 

2.7 

-0.071 

0.01 

-0.042 

0 

1 

0.885 

0.813 

0.903 

0.932 

0.903 

0.616 

1 

0.823 

0.823 

0.854 

0.907 

1.059 

0.977 

1.088 

0.789 

2 

-3 

0 

3.5 

-0.031 

0.012 

0.006 

-1 

-1 

0.727 

0.881 

0.637 

0.856 

0.936 

0.672 

1.09 

0.715 

0.686 

1.071 

0.7 

0.825 

0.981 

0.928 

0.692 

2 

2 

2 

-1.8 

-0.066 

0.016 

0.024 

-1 

0 

0.875 

0.858 

1.061 

0.987 

0.909 

1.043 

0.612 

0.901 

0.94 

0.768 

0.909 

0.817 

0.752 

1.026 

0.606 

1 

-3 

2 

-0.9 

-0.036 

0.001 

-0.035 

-1 

1 

0.704 

0.983 

0.854 

0.694 

1.002 

1.032 

0.827 

0.862 

0.868 

0.696 

0.838 

0.61 

0.842 

1.092 

0.827 

0 

1 

0 

-0.4 

-0.087 

:  ::e 

0.012 

0 

2 

1.047 

0.7 

0.866 

0.955 

0.641 

0.801 

0.86 

0.688 

0.715 

0.643 

0.989 

0.848 

0.797 

0.991 

0.745 

0 

-1 

-1 

0.1 

-0.09 

0.036 

0.018 

-1 

-1 

0.621 

0.932 

0.819 

0.842 

0.639 

0.719 

0.963 

0.848 

0.666 

0.645 

1.092 

0.704 

0.653 

1.016 

0.75 

1 

1 

2 

3.9 

0.031 

-0.016 

0.021 

-1 

0 

0.715 

0.913 

1.055 

0.963 

0.811 

0.614 

1.071 

0.889 

0.942 

0.709 

0.895 

1.012 

1.065 

1.079 

1.069 
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0 

-3 

1 

1.3 

0.027 

-0.018 

-0.038 

-1 

0 

1.061 

0.801 

0.832 

0.791 

0.758 

0.661 

0.922 

1.026 

1.096 

0.827 

0.942 

1.098 

0.909 

1.049 

0.887 

0 

1 

1 

1.6 

0.034 

-0.029 

-0.029 

-2 

1 

0.659 

1.092 

0.924 

0.977 

0.883 

0.991 

0.788 

0.608 

0.756 

0.602 

0.998 

0.928 

1.012 

0.854 

0.948 

0 

0 

0 

1.7 

0.042 

-0.042 

0.003 

-1 

-1 

1.082 

0.741 

0.651 

0.621 

1.051 

1.08 

0.774 

0.768 

0.846 

0.834 

1.082 

0.92 

1.092 

0.985 

0.94 

0 

1 

3 

0 

0.003 

-0.046 

0.017 

-2 

-1 

0.641 

1.028 

0.914 

1.014 

1.08 

0.705 

0.893 

0.995 

1.079 

0.973 

0.6 

0.77 

0.725 

0.873 

1.041 

0 

-1 

1 

0.9 

0.043 

-0.032 

0.002 

-1 

2 

0.965 

0.633 

0.823 

0.633 

0.809 

0.803 

0.944 

1.028 

0.899 

1.084 

0.735 

0.7 

0.627 

1.055 

1.026 

0 

4 

1 

0.6 

-0.012 

-0.023 

-0.049 

0 

0 

0.78 

0.975 

1.024 

1.065 

0.854 

1.036 

0.668 

0.774 

0.674 

0.926 

0.764 

0.643 

0.704 

1.028 

0.979 

-2 

0 

1 

-1.1 

0.002 

-0.016 

-0.027 

-2 

0 

0.895 

0.817 

Z.5Z- 

0.838 

0.711 

0.946 

0.616 

0.789 

0.698 

1.075 

0.704 

0.86 

0.782 

1.059 

0.996 

0 

1 

2 

3.3 

-0.097 

-0.05 

-0.014 

0 

-2 

0.916 

1.022 

0.971 

1.039 

0.68 

0.852 

0.94 

1.088 

0.934 

0.957 

1.039 

1.088 

0.926 

0.782 

0.69 

0 

-1 

3 

3.1 

-0.033 

-0.039 

-0.023 

-2 

0 

0.666 

0.602 

0.709 

0.752 

0.616 

0.942 

1.022 

0.899 

0.897 

0.891 

1.008 

0.959 

1.039 

0.719 

0.612 

-2 

2 

1 

3.3 

-0.04 

-0.031 

-0.024 

0 

0 

0.942 

0.86 

1.032 

0.903 

1.053 

0.762 

0.711 

0.69 

0.616 

0.815 

0.932 

1.036 

0.991 

0.723 

0.748 

0 

-4 

0 

3.7 

-0.043 

-0.004 

-0.034 

-1 

0 

0.834 

0.692 

0.67 

0.817 

1.038 

0.655 

0.618 

0.67 

0.784 

0.989 

0.946 

0.961 

0.852 

0.616 

0.829 

-2 

3 

3 

-0.1 

-0.05 

-0.027 

-0.005 

0 

-2 

0.871 

0.871 

1 

0.989 

0.954 

1 

1.053 

1.082 

0.922 

0.713 

0.795 

0.666 

0.815 

0.614 

0.754 

-1 

-4 

2 

-1.7 

-0.027 

-0.007 

-0.021 

-1 

2 

0.766 

0.737 

0.756 

0.788 

0.873 

0.858 

1.047 

0.885 

1.08 

0.823 

0.614 

0.698 

0.825 

0.727 

0.641 

-1 

3 

-1 

0.5 

-0.034 

-0.041 

0.004 

-2 

2 

1.043 

1.096 

1.041 

0.893 

0.635 

0.805 

0.795 

0.707 

0.696 

0.756 

0.78 

0.711 

0.623 

0.678 

0.602 

-1 

-3 

-1 

-0.1 

-0.078 

-0.043 

-0.037 

0 

-2 

0.821 

0.707 

0.743 

0.643 

0.788 

0.746 

0.725 

0.68 

0.848 

0.692 

0.602 

0.752 

0.731 

0.762 

0.877 

1 

0 

1 

1 

-0.025 

0 

0.025 

0 

0 

0.85 

0.85 

0.85 

0.85 

0.85 

0.85 

0.85 

0.85 

0.85 

0.85 

0.85 

0.85 

0.85 

0.85 

0.85 

1 

-3 

1 

0.3 

-0.044 

-0.04 

-0.05 

-2 

1 

0.981 

0.907 

1.069 

1.008 

0.93 

0.903 

0.877 

1.071 

1.082 

1.014 

0.961 

0.979 

0.954 

0.913 

1.004 

2 

1 

0 

-1.7 

-0.058 

-0.003 

-0.046 

-1 

0 

0.788 

0.729 

0.6 

0.657 

0.844 

1.088 

1.069 

1.032 

1.098 

0.924 

0.811 

0.981 

0.893 

0.84 

0.975 

2 

-1 

3 

0.6 

-0.094 

-0.031 

-0.041 

-2 

-1 

0.998 

0.977 

1.036 

1.028 

0.631 

0.684 

0.817 

0.635 

0.678 

0.979 

1.059 

1.024 

1.082 

0.946 

1.092 

1 

3 

1 

0.2 

-0.06 

-0.025 

-0.032 

-2 

1 

0.631 

0.709 

0.618 

0.623 

0.778 

0.809 

0.705 

0.6 

0.711 

1.092 

1.03 

0.995 

0.877 

0.983 

0.903 

1 

0 

1 

1.6 

-0.089 

-0.029 

-0.013 

-1 

2 

1.057 

1.073 

0.94 

1.061 

0.848 

1.02 

1.026 

1.043 

0.961 

0.635 

0.772 

0.678 

0.67 

0.954 

0.93 

1 

1 

-1 

3 

-0.079 

-0.041 

-0.01 

-2 

-1 

0.68 

0.719 

0.68 

0.746 

0.643 

0.887 

Z.3Z3 

0.879 

0.914 

0.655 

0.664 

0.647 

0.61 

0.815 

1.084 

3 

-1 

1 

3.3 

-0.027 

-0.002 

-0.015 

0 

-1 

1.098 

0.965 

0.946 

0.94 

1.028 

0.645 

0.643 

0.77 

0.61 

0.702 

0.856 

0.62 

0.692 

0.948 

0.899 

1 

4 

2 

1.6 

-0.089 

-0.024 

-0.005 

-1 

0 

0.868 

0.834 

0.696 

0.774 

0.887 

0.647 

0.735 

0.684 

0.793 

0.76 

0.676 

0.621 

0.639 

0.987 

0.965 

3 

0 

-1 

0.2 

-0.008 

-0.029 

-0.011 

-2 

1 

0.627 

0.985 

0.871 

0.885 

1.008 

0.772 

0.897 

0.952 

0.93 

0.791 

1.006 

1.065 

0.987 

0.666 

0.651 

1 

3 

0 

-0.3 

0.044 

-0.011 

0.015 

0 

-2 

0.899 

0.758 

0.721 

0.625 

1.034 

0.821 

0.879 

1.045 

0.973 

0.688 

0.852 

0.963 

0.891 

0.842 

0.713 

2 

-3 

1 

-1.8 

-0.016 

0 

-0.046 

0 

-2 

0.778 

0.995 

0.995 

0.899 

0.776 

1.049 

0.715 

0.84 

0.836 

0.664 

0.885 

1.094 

0.838 

0.61 

0.731 

3 

3 

3 

-0.4 

-0.005 

-0.017 

-0.015 

0 

0 

0.881 

0.854 

0.61 

0.782 

0.741 

0.883 

0.801 

0.7 

0.682 

0.793 

0.969 

0.944 

0.959 

0.707 

0.729 

2 

-4 

0 

3.6 

-0.011 

-0.03 

0.003 

-1 

1 

0.717 

0.996 

1.079 

0.864 

0.694 

0.766 

0.971 

1.061 

0.998 

0.881 

0.748 

0.68 

0.77 

0.76 

0.661 

3 

2 

0 

3.2 

0.026 

-0.03 

-0.048 

0 

-1 

0.924 

0.711 

0.764 

0.77 

0.799 

0.666 

0.889 

0.928 

0.946 

0.879 

0.842 

0.616 

0.793 

0.668 

0.627 

2 

-1 

3 

1.1 

0.032 

-0.003 

-0.014 

0 

-1 

0.817 

0.967 

1.077 

1.026 

1.045 

0.948 

0.68 

0.621 

0.829 

1.039 

0.68 

0.829 

0.817 

0.875 

0.647 

1 

2 

2 

3.6 

-0.01 

-0.022 

-0.041 

-1 

2 

1.094 

0.836 

0.713 

0.811 

1.012 

0.916 

0.746 

0.735 

0.821 

0.864 

0.707 

0.774 

0.645 

0.688 

0.6 
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2 

0 

0 

-0.7 

-0.085 

0.013 

0.004 

-1 

1 

0.846 

0.774 

1.039 

0.936 

0.934 

0.932 

0.686 

1.024 

1.045 

0.942 

0.672 

1.039 

0.897 

0.78 

0.821 

1 

2 

0 

0.3 

-0.03 

0.022 

-0.028 

-1 

-1 

0.647 

0.979 

0.795 

0.786 

0.862 

1.014 

0.657 

1.008 

1.024 

0.858 

0.803 

0.95 

0.944 

0.62 

0.944 

2 

-2 

1 

1 

-0.072 

0.046 

-0.031 

-1 

-1 

0.979 

0.739 

1.022 

1.051 

0.754 

0.836 

0.907 

0.717 

0.731 

1.094 

0.627 

1.049 

1.01 

0.631 

0.993 

2 

2 

2 

0 

-0.062 

0.023 

-0.017 

-2 

0 

0.698 

1.09 

0.672 

0.696 

0.713 

0.676 

0.957 

0.745 

0.817 

i  ::s 

0.62 

1.047 

1.086 

0.768 

1.036 

1 

0 

0 

2.8 

-0.069 

0.043 

-0.048 

-2 

1 

1.086 

0.702 

1.045 

1.016 

0.637 

0.952 

0.641 

1.055 

0.895 

0.707 

1.026 

0.754 

0.633 

0.786 

0.918 

1 

1 

1 

2.8 

-0.086 

0.036 

-0.03 

-1 

-1 

0.729 

1.079 

0.625 

0.608 

0.858 

0.963 

0.891 

0.979 

0.977 

0.737 

1.061 

0.608 

0.727 

0.807 

0.913 

1 

-2 

1 

1.2 

-0.028 

0.002 

-0.017 

-1 

-1 

0.991 

0.653 

1.014 

0.922 

0.952 

0.608 

1.049 

0.661 

0.672 

0.871 

0.963 

0.692 

0.737 

0.637 

0.959 

2 

1 

2 

2.3 

-0.061 

0.043 

-0.022 

-1 

0 

0.858 

0.848 

0.733 

0.729 

0.973 

0.76 

LI 

0.614 

0.631 

0.698 

0.926 

0.762 

0.778 

0.832 

0.87 

2 

0 

0 

-1.4 

0.018 

0.011 

0.02 

-2 

2 

0.789 

0.684 

0.954 

1.073 

0.895 

0.717 

0.717 

0.94 

0.866 

0.7 

0.846 

0.885 

1.041 

1.024 

0.614 

1 

2 

-1 

0.8 

-0.009 

0.046 

-0.003 

-2 

-1 

0.993 

0.973 

0.858 

0.62 

0.95 

0.844 

0.842 

0.903 

1.036 

0.883 

0.663 

0.868 

1.08 

0.83 

0.723 

3 

-2 

1 

-0.9 

-0.024 

0.006 

0.009 

-1 

-1 

0.764 

0.825 

0.955 

0.871 

0.823 

0.889 

1.036 

0.723 

0.639 

0.746 

0.623 

0.957 

0.873 

1.01 

0.672 

2 

3 

2 

-0.5 

0 

0.013 

-0.004 

0 

0 

0.838 

0.889 

0.817 

0.698 

0.61 

0.979 

0.948 

0.819 

0.645 

0.704 

0.809 

1.026 

1.016 

1.075 

0.633 

2 

-3 

-1 

2.8 

0.02 

0.009 

-0.025 

-1 

0 

0.725 

0.713 

0.985 

0.911 

0.676 

0.698 

0.832 

0.836 

1.063 

0.969 

0.877 

0.746 

0.762 

0.967 

0.719 

2 

2 

1 

3.4 

0.02 

0.034 

-0.013 

0 

-2 

1.022 

0.885 

0.702 

0.731 

0.78 

0.907 

0.655 

0.934 

1.075 

0.868 

0.84 

0.784 

0.733 

1.077 

0.694 

1 

-3 

2 

1.1 

-0.021 

:  G3S 

-0.001 

-2 

-1 

0.756 

0.803 

0.895 

1.071 

1.039 

1.008 

1.024 

0.725 

0.782 

0.922 

0.987 

0.76 

0.641 

0.996 

0.682 

2 

0 

2 

2.5 

0.008 

0.01 

0.019 

-2 

1 

0.889 

1.03 

0.782 

0.651 

0.991 

1.067 

0.887 

0.643 

0.7 

1.032 

0.903 

0.819 

0.631 

0.942 

0.78 

1 

-1 

0 

0.8 

-0.046 

-0.05 

0.085 

-1 

1 

1.03 

1.086 

0.809 

0.887 

1.03 

1.01 

1.006 

0.75 

1.092 

1.024 

0.995 

0.745 

0.998 

1.043 

0.666 

1 

0 

0 

-0.1 

-0.078 

-0.047 

0.03 

0 

-2 

0.793 

0.618 

1.018 

0.725 

0.911 

0.987 

1.073 

0.623 

C.SS5 

1.077 

0.985 

0.854 

0.881 

0.963 

0.856 

2 

-3 

2 

0.4 

-0.081 

-0.044 

0.072 

0 

0 

0.995 

1.012 

0.692 

1.055 

0.825 

0.678 

0.604 

0.922 

0.827 

0.977 

0.967 

0.684 

0.916 

1.071 

0.676 

2 

2 

1 

-0.2 

-0.067 

-0.038 

0.062 

-2 

0 

0.674 

0.625 

0.993 

0.805 

0.829 

0.727 

0.606 

0.963 

0.717 

1.079 

1.067 

0.748 

0.946 

0.887 

0.67 

1 

-1 

1 

3 

-0.098 

-0.025 

0.069 

-1 

0 

1.077 

0.946 

0.627 

0.873 

0.866 

1.071 

1.002 

0.741 

0.981 

0.739 

0.653 

1.032 

0.668 

1.008 

0.758 

1 

2 

0 

25 

-0.08 

-0.023 

0.086 

0 

0 

0.616 

0.676 

0.983 

0.678 

0.819 

1.026 

0.979 

0.743 

0.881 

0.719 

0.643 

1.03 

0.75 

1.065 

0.862 

1 

0 

1 

2 

-0.067 

-0.027 

0.028 

-1 

-1 

1.036 

0.944 

0.704 

0.938 

1.004 

0.67 

0.758 

1.084 

0.838 

0.62 

0.612 

0.899 

0.864 

0.895 

0.811 

1 

1 

2 

1.2 

-0.072 

-0.02 

0.061 

-1 

0 

0.723 

0.668 

0.852 

0.733 

1.043 

0.618 

0.709 

0.944 

0.647 

0.764 

0.655 

1.077 

0.707 

1 

0.702 

2 

-2 

-1 

0.2 

-0.02 

-0.024 

0.068 

-1 

1 

0.75 

0.924 

0.788 

0.924 

1.047 

0.782 

0.918 

0.746 

0.903 

0.852 

1.096 

0.723 

0.84 

0.698 

0.881 

2 

3 

0 

-1.1 

0.003 

-0.007 

0.041 

0 

-2 

0.852 

0.778 

0.989 

0.756 

1.016 

0.745 

1.08 

0.829 

1.094 

0.825 

1.016 

0.821 

0.879 

0.659 

0.86 

2 

-4 

2 

-1.2 

-0.009 

-0.048 

0.026 

-2 

-2 

0.772 

1.006 

0.752 

0.891 

0.7 

0.944 

0.733 

0.868 

0.825 

0.715 

1.014 

0.727 

1.079 

0.655 

1.063 

3 

3 

2 

-0.7 

0.004 

-0.027 

0.05 

-1 

1 

0.891 

0.612 

0.838 

0.612 

0.704 

0.832 

0.754 

0.924 

0.789 

0.647 

1.01 

0.764 

0.868 

0.741 

1.014 

2 

-3 

0 

3.9 

0.025 

-0.014 

0.07 

0 

1 

0.746 

1.077 

0.684 

1.082 

0.743 

0.866 

1.098 

0.752 

0.86 

0.948 

0.659 

0.914 

0.811 

0.661 

0.967 

2 

2 

0 

3.2 

0.013 

-0.049 

0.07 

-2 

-1 

1.065 

0.764 

0.975 

0.86 

0.651 

0.795 

0.924 

0.786 

0.889 

0.914 

0.647 

0.873 

0.625 

0.6 

1.002 

2 

-1 

2 

2.7 

0.001 

-0.026 

0.029 

0 

-1 

0.87 

1.038 

0.727 

0.866 

1.077 

0.996 

0.633 

1.004 

0.635 

0.887 

0.745 

0.911 

0.829 

0.663 

0.905 

1 

3 

2 

2.9 

-0.019 

-0.044 

0.058 

0 

1 

0.969 

0.659 

0.84 

0.784 

1.079 

1.051 

0.789 

0.967 

0.844 

0.83 

0.786 

1.067 

0.758 

0.627 

1.022 
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3 

-2 

0 

-0.4 

-0.1 

0.014 

0.044 

0 

2 

1.063 

0.664 

0.741 

1.047 

0.967 

1.061 

0.661 

0.805 

0.92 

0.952 

0.719 

0.69 

1.045 

0.819 

0.737 

2 

4 

-1 

0.6 

-0.096 

0.035 

0.058 

-2 

0 

0.6 

0.957 

0.928 

0.854 

1.096 

1.077 

0.692 

0.604 

0.975 

1.026 

0.723 

0.793 

1.02 

0.686 

0.629 

3 

-1 

2 

-1.9 

-0.091 

0.037 

0.094 

-1 

-2 

1.016 

0.69 

0.762 

0.901 

0.786 

0.643 

0.961 

0.998 

0.633 

1.034 

0.682 

0.659 

0.961 

0.653 

0.705 

1 

4 

3 

-0.6 

-0.082 

0.028 

0.06 

-1 

1 

0.608 

1.018 

1.094 

0.655 

0.784 

0.6 

0.952 

0.893 

0.807 

0.913 

0.688 

0.6 

0.854 

0.68 

0.815 

2 

-4 

0 

1.9 

-0.063 

0.048 

0.089 

0 

0 

0.959 

0.639 

0.674 

0.946 

0.608 

1.012 

0.702 

0.663 

0.991 

0.745 

0.983 

0.987 

0.78 

0.735 

0.793 

3 

2 

1 

3.9 

-0.06 

0.037 

0.079 

-1 

-1 

0.739 

1.045 

1.088 

0.832 

0.76 

1.038 

0.69 

0.842 

1.006 

0.614 

1.071 

1.061 

0.651 

0.778 

0.832 

1 

-2 

3 

2.6 

-0.065 

0.028 

0.027 

-2 

-1 

0.987 

0.645 

0.657 

0.996 

1.041 

0.786 

1.079 

1.063 

0.774 

0.627 

1.004 

0.991 

0.696 

0.823 

0.635 

2 

1 

2 

3.6 

-0.055 

0.032 

0.089 

-1 

2 

0.737 

0.928 

0.879 

0.6 

0.928 

0.823 

1.075 

1.034 

0.748 

0.61 

1.034 

0.862 

0.87 

0.856 

0.762 

1 

-2 

0 

-2 

0.011 

0.004 

0.042 

-1 

1 

0.795 

0.823 

0.635 

1.024 

1.032 

0.739 

0.645 

0.788 

1.073 

0.612 

0.819 

0.805 

1.053 

0.924 

0.991 

2 

2 

0 

-1.8 

-0.015 

0.018 

0.094 

-1 

-1 

0.932 

0.901 

0.911 

0.743 

1.067 

0.602 

0.797 

0.834 

1.032 

0.741 

0.832 

0.627 

0.914 

0.957 

0.866 

2 

-2 

2 

-1.6 

0.046 

0.011 

0.034 

-1 

-1 

0.7 

0.77 

0.813 

1.059 

0.602 

0.881 

0.838 

1.069 

0.69 

0.604 

0.625 

0.682 

1.022 

0.971 

1.045 

2 

2 

2 

-1.3 

0.015 

0.02 

0.045 

-2 

1 

1.08 

0.94 

0.899 

0.702 

0.6 

0.95 

0.836 

0.907 

0.752 

0.784 

0.75 

0.811 

1.026 

1.067 

1.08 

2 

-4 

-1 

2.5 

-0.003 

0.034 

0.039 

-1 

2 

0.604 

0.746 

0.688 

1.036 

0.729 

0.659 

0.854 

0.782 

1.041 

1.049 

0.94 

0.877 

0.602 

0.995 

0.924 

3 

3 

-1 

2.4 

0.048 

0.025 

0.076 

-1 

-2 

0.93 

1.049 

0.907 

0.803 

0.664 

0.778 

0.727 

0.627 

0.891 

1.1 

0.922 

0.836 

0.666 

0.934 

1.096 

3 

-4 

3 

2.6 

0.014 

0.017 

0.082 

-1 

-2 

0.645 

0.762 

0.825 

1.063 

0.969 

1.096 

0.955 

1.014 

0.629 

0.889 

0.821 

1.086 

0.698 

1.082 

1.032 

3 

4 

3 

2.2 

0.041 

0.004 

0.04 

-2 

2 

1.01 

0.914 

1.057 

0.793 

0.897 

0.93 

0.977 

0.87 

0.737 

0.899 

0.944 

0.913 

0.764 

0.905 

0.875 

3 

-1 

0 

-0.5 

-0.029 

-0.021 

-0.031 

2 

1 

1.032 

0.934 

1.01 

0.604 

1.082 

0.873 

1.065 

1.098 

0.67 

1.043 

1.051 

0.918 

0.776 

0.926 

1.016 

1 

3 

0 

-0.9 

-0.088 

-0.008 

-0.027 

2 

-1 

0.813 

0.78 

0.647 

1.094 

1.055 

0.989 

1.016 

1.094 

0.688 

1.063 

0.934 

0.905 

0.705 

0.916 

0.858 

2 

-2 

2 

-1 

-0.079 

-0.007 

-0.034 

2 

-1 

0.901 

1.094 

1.059 

0.631 

0.782 

0.61 

0.766 

0.711 

1.086 

0.905 

0.866 

1.045 

0.721 

1.053 

0.901 

2 

2 

3 

-1.4 

-0.047 

-0.047 

-0.042 

2 

1 

0.655 

0.83 

0.629 

1.03 

0.762 

0.764 

0.856 

0.649 

1.1 

0.799 

0.918 

0.983 

0.7 

0.879 

0.914 

2 

-3 

0 

2.2 

-0.033 

-0.033 

-0.003 

1 

1 

0.926 

0.893 

1.047 

0.735 

0.674 

0.985 

0.969 

0.969 

0.735 

0.805 

0.635 

0.625 

1.051 

0.952 

0.922 

2 

3 

0 

2.8 

-0.048 

-0.008 

-0.043 

1 

-1 

0.84 

0.698 

0.739 

0.973 

0.756 

1.094 

1.063 

0.875 

0.762 

0.672 

0.789 

0.672 

1.071 

1.061 

0.909 

2 

-2 

2 

1.8 

-0.099 

-0.019 

0.012 

1 

-2 

1.038 

0.911 

0.916 

0.717 

1.073 

0.743 

0.848 

0.653 

0.959 

0.639 

0.788 

0.686 

1.1 

0.969 

0.936 

2 

3 

3 

3.5 

-0.077 

-0.021 

-0.002 

1 

1 

0.625 

0.686 

0.799 

0.952 

0.977 

0.838 

0.768 

0.713 

0.967 

0.772 

0.725 

0.702 

1.063 

0.998 

1.079 

2 

■A 

1 

0.9 

0.014 

-0.021 

-0.047 

1 

1 

0.686 

0.948 

0.87 

0.819 

0.965 

0.641 

0.87 

0.891 

0.713 

0.618 

0.836 

1.006 

0.612 

0.803 

0.739 

3 

3 

0 

-1.5 

0.023 

-0.012 

0.017 

0 

0 

0.955 

0.795 

0.608 

0.979 

0.993 

0.653 

1.092 

0.916 

0.705 

0.78 

0.901 

0.893 

0.745 

0.77 

0.774 

1 

0 

3 

-1.2 

0.024 

-0.045 

0.01 

2 

-1 

0.629 

0.903 

0.889 

0.7 

0.682 

0.967 

0.77 

0.754 

0.909 

0.653 

0.938 

0.856 

0.604 

0.838 

0.663 

1 

1 

2 

0.1 

0.034 

-0.013 

0.007 

1 

2 

1.004 

0.616 

0.807 

1.09 

0.774 

0.909 

0.784 

0.625 

1.039 

0.659 

0.83 

1.004 

0.799 

0.711 

0.715 

1 

-3 

1 

1.3 

0.005 

-0.001 

0 

1 

2 

0.688 

1.069 

0.864 

0.614 

0.721 

0.635 

0.871 

1.006 

0.623 

1.059 

0.825 

0.731 

0.971 

0.649 

0.623 

2 

1 

-1 

1.9 

0.021 

-0.04 

0.023 

2 

0 

0.952 

0.856 

0.735 

0.942 

0.614 

0.78 

0.928 

0.887 

0.772 

1.022 

0.621 

0.809 

1.094 

0.801 

0.657 

3 

0 

2 

4 

-0.004 

-0.009 

0.016 

1 

-1 

0.672 

0.827 

1.034 

0.616 

0.948 

1.004 

0.672 

0.78 

1.047 

1.018 

0.721 

0.799 

0.844 

0.891 

0.746 

1 

3 

2 

3.1 

0.037 

-0.005 

-0.028 

2 

1 

0.918 

0.784 

0.827 

0.985 

0.83 

0.971 

0.866 

0.764 

0.957 

0.897 

0.729 

0.631 

0.989 

0.756 

0.625 
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-1 

1 

-0.3 

-0.056 

0.002 

0.001 

1 

2 

0.873 

0.641 

0.932 

0.666 

0.885 

1.079 

0.75 

1.049 

0.684 

0.971 

0.727 

0.942 

0.664 

0.67 

1.055 

0 

-1 

0.3 

-0.037 

0.042 

-0.001 

1 

0 

0.711 

0.891 

0.78 

0.95 

1.088 

0.87 

0.713 

1.053 

0.641 

0.938 

0.616 

0.924 

0.663 

0.829 

1.061 

0 

1 

-1.7 

-0.036 

0.036 

-0.018 

1 

0 

1.039 

0.805 

1.08 

0.707 

0.768 

0.774 

1.038 

0.827 

1.051 

0.893 

0.807 

0.985 

0.682 

0.694 

0.897 

3 

1 

0.3 

-0.095 

0.015 

-0.038 

1 

1 

0.803 

1.043 

0.698 

1.018 

0.801 

0.702 

0.993 

0.846 

0.996 

0.946 

0.827 

1.063 

0.616 

0.789 

0.983 

-4 

0 

1 

-0.075 

0.005 

-0.045 

2 

1 

0.877 

0.791 

1.006 

0.663 

0.868 

0.893 

0.696 

0.971 

0.651 

0.631 

0.948 

0.629 

0.965 

0.725 

0.883 

1 

1 

3.4 

-0.037 

0.015 

-0.024 

1 

-1 

0.692 

0.862 

0.616 

1.098 

0.821 

0.993 

0.688 

0.996 

0.707 

0.797 

0.959 

0.707 

1.024 

0.682 

1.02 

-2 

3 

1.5 

-0.054 

0.049 

-0.009 

1 

0 

0.946 

0.629 

0.952 

0.68 

1.022 

0.829 

0.914 

0.698 

1.02 

0.856 

1.049 

0.768 

1.043 

0.75 

1.082 

1 

2 

1.3 

-0.057 

0.035 

0.002 

1 

2 

0.733 

1.08 

0.758 

0.995 

0.893 

0.694 

0.901 

0.682 

1.088 

0.67 

0.871 

0.725 

0.934 

0.721 

1.057 

-2 

0 

-0.2 

0.007 

0.026 

-0.049 

0 

1 

0.786 

0.75 

1.051 

0.78 

1.071 

0.741 

0.623 

0.844 

0.721 

0.84 

0.645 

1.051 

0.76 

1.057 

0.768 

2 

1 

0.5 

-0.007 

0.032 

-0.033 

1 

-1 

0.94 

1.02 

0.614 

0.877 

0.864 

0.631 

0.762 

1.067 

0.858 

0.789 

0.77 

1.038 

0.805 

1.096 

0.766 

-3 

2 

0.6 

0.042 

0.033 

-0.021 

0 

-1 

0.791 

0.661 

0.897 

0.766 

0.745 

0.924 

0.936 

0.62 

0.936 

0.717 

0.606 

0.87 

0.717 

0.909 

0.727 

1 

2 

-1.8 

-0.006 

0.039 

0.019 

1 

1 

1.024 

1.034 

0.719 

0.905 

0.725 

0.965 

0.981 

0.678 

0.913 

0.838 

0.743 

0.965 

0.748 

0.866 

0.61 

-1 

0 

3 

-0.024 

0.02 

-0.016 

1 

0 

0.649 

0.696 

0.885 

0.741 

0.739 

0.84 

0.67 

1.036 

0.799 

0.93 

1.069 

0.813 

0.887 

0.961 

0.653 

4 

1 

15 

0.035 

0.03 

0.014 

1 

-1 

0.844 

0.969 

0.737 

1.041 

0.702 

0.846 

0.815 

1.09 

0.795 

1.038 

1.022 

0.657 

0.932 

1.004 

0.688 

-2 

1 

2.1 

-0.012 

0.014 

-0.037 

1 

-2 

0.836 

0.635 

1.067 

0.83 

0.86 

1.018 

1.041 

0.795 

0.954 

0.965 

0.916 

0.645 

0.913 

1.094 

0.836 

1 

3 

2.3 

-0.018 

0.041 

-0.04 

0 

0 

0.903 

0.879 

0.663 

0.928 

0.981 

0.901 

0.996 

0.727 

1.057 

0.934 

0.899 

0.604 

0.911 

0.889 

0.772 

-3 

0 

-2 

-0.049 

-0.038 

0.056 

2 

0 

0.893 

1.053 

0.791 

0.825 

0.963 

1.063 

1.034 

0.659 

0.75 

0.967 

0.924 

0.729 

0.743 

1.036 

0.809 

3 

1 

-1.3 

-0.051 

-0.034 

0.037 

0 

-1 

0.762 

0.674 

0.926 

1.032 

1.01 

0.885 

1.086 

0.817 

0.729 

0.95 

0.883 

0.778 

0.795 

0.883 

0.807 

-2 

3 

-0.9 

-0.068 

-0.039 

0.036 

1 

-2 

1.049 

1.051 

0.602 

0.69 

0.715 

0.789 

0.649 

0.942 

0.932 

1.016 

1.032 

0.834 

0.715 

0.864 

0.7 

1 

3 

0.7 

-0.088 

-0.045 

0.095 

1 

2 

0.694 

0.6 

0.856 

0.961 

0.625 

0.725 

0.682 

0.938 

1.043 

1.02 

0.991 

0.766 

0.643 

0.955 

0.846 

-2 

-1 

2.6 

-0.095 

-0.019 

0.075 

1 

1 

1.018 

1.063 

0.778 

0.647 

0.717 

0.969 

0.959 

0.791 

0.692 

0.69 

0.698 

1.082 

1.014 

0.907 

0.784 

0 

-1 

1.4 

-0.074 

045 

0.037 

2 

-2 

0.612 

0.748 

0.977 

0.981 

0.705 

0.913 

0.895 

0.719 

0.813 

0.725 

0.61 

0.909 

0.928 

0.852 

:.e-9 

-4 

2 

35 

-0.059 

-0.009 

0.054 

0 

0 

0.957 

0.868 

0.731 

0.709 

0.905 

0.723 

0.639 

0.897 

1.038 

0.782 

0.637 

0.903 

0.899 

0.936 

0.848 

3 

1 

3.6 

-0.048 

-0.018 

0.057 

1 

1 

0.752 

0.745 

0.934 

0.852 

1.014 

0.625 

0.78 

0.991 

0.955 

0.705 

0.746 

0.967 

1.028 

1.098 

0.711 

-2 

-1 

0 

0.049 

:-e 

0.057 

1 

1 

0.633 

0.918 

0.77 

0.821 

1.094 

0.825 

0.875 

0.739 

0.776 

0.633 

1.043 

0.803 

0.694 

0.799 

0.895 

1 

0 

-0.1 

0.033 

-0.005 

0.043 

0 

-2 

1.09 

0.643 

0.95 

1.012 

0.907 

0.627 

0.881 

0.641 

0.87 

0.649 

0.887 

0.842 

0.834 

0.797 

0.957 

-4 

2 

-0.7 

0.021 

-0.01 

0.092 

0 

-1 

0.815 

0.887 

0.797 

0.768 

0.797 

1.084 

0.7 

0.877 

0.877 

0.846 

0.793 

0.641 

0.723 

0.612 

0.911 

3 

2 

-15 

-0.019 

-0.012 

0.044 

1 

1 

0.973 

0.819 

1.063 

0.844 

0.764 

1.028 

0.61 

0.985 

1.014 

0.629 

1 

0.875 

0.719 

0.772 

1.008 

-2 

0 

3.8 

0.016 

-0.016 

0.026 

1 

0 

0.825 

0.842 

0.639 

0.713 

0.791 

0.657 

1.088 

0.799 

0.76 

0.932 

0.791 

0.883 

0.948 

0.674 

1.094 

3 

0 

2.9 

-0.014 

-0.001 

0.085 

1 

-1 

0.996 

0.717 

0.846 

1.006 

0.698 

0.668 

0.873 

0.838 

0.832 

1.004 

0.862 

1.09 

0.858 

0.608 

0.873 

-1 

2 

2.4 

0.037 

-0.006 

0.038 

0 

-2 

0.653 

1 

0.834 

0.745 

1.059 

0.899 

0.84 

1.012 

0.985 

1.057 

0.711 

0.852 

0.903 

0.709 

0.955 

1 

3 

1.9 

0.04 

-0.036 

0.032 

1 

1 

1.079 

0.768 

0.881 

0.858 

1.061 

0.981 

0.737 

0.852 

1.034 

1.055 

0.608 

0.996 

1.047 

0.684 

0.95 
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2 

-1 

0 

-1.9 

-0.081 

0.016 

0.029 

1 

0 

0.985 

0.788 

0.645 

0.737 

0.889 

1.086 

0.629 

0.811 

0.758 

0.991 

0.805 

0.688 

0.635 

0.621 

0.631 

1 

3 

1 

0.7 

-0.077 

0.018 

0.088 

1 

0 

0.639 

0.899 

0.868 

0.909 

0.942 

1.039 

0.778 

0.674 

0.604 

0.873 

0.758 

0.602 

0.791 

0.651 

0.813 

1 

-1 

1 

0.4 

-0.084 

0.029 

0.079 

2 

-1 

1.041 

0.608 

0.776 

0.723 

0.817 

0.709 

0.913 

1.092 

0.944 

1.098 

0.702 

0.772 

0.688 

0.846 

0.752 

1 

0 

2 

0.3 

-0.092 

0.042 

0.047 

1 

1 

0.618 

0.959 

1.049 

1.079 

0.649 

0.62 

0.926 

0.932 

0.854 

0.866 

0.618 

0.78 

0.608 

0.715 

0.76 

1 

-1 

-1 

2 

-0.053 

0.046 

0.033 

2 

1 

1.059 

0.672 

0.786 

0.686 

0.62 

0.995 

0.807 

0.705 

0.621 

0.727 

1.1 

0.93 

0.975 

0.827 

0.659 

1 

1 

1 

1.1 

-0.093 

0.032 

0.048 

1 

-2 

0.735 

1.067 

0.877 

1.067 

0.891 

0.897 

0.756 

0.672 

0.801 

0.616 

0.965 

1 

1.073 

0.645 

0.674 

1 

-4 

1 

1.4 

-0.038 

0.023 

0.099 

0 

0 

0.92 

0.725 

0.676 

0.635 

0.846 

0.664 

1.032 

0.926 

1.026 

0.774 

0.936 

1.057 

0.996 

0.672 

0.721 

3 

0 

1 

3.1 

-0.052 

0.016 

0.077 

2 

0 

0.805 

0.883 

1.096 

0.862 

0.989 

0.754 

1.084 

0.911 

1.002 

0.625 

0.996 

0.84 

0.918 

0.641 

0.704 

1 

-1 

0 

-1.3 

0.047 

0.05 

0.064 

0 

2 

0.784 

0.678 

0.729 

0.661 

1.02 

0.848 

0.76 

0.612 

0.766 

0.743 

0.661 

0.612 

0.774 

0.918 

1.01 

1 

1 

-1 

-1.1 

-0.017 

0.039 

0.073 

2 

0 

1.034 

1.098 

0.991 

0.948 

1.084 

0.758 

0.678 

0.801 

0.803 

0.809 

0.692 

0.741 

0.661 

0.981 

1.088 

3 

-2 

1 

-1.3 

-0.01 

0.031 

0.074 

0 

0 

0.758 

0.84 

0.668 

0.797 

0.647 

0.938 

0.989 

1.01 

1.084 

0.885 

0.768 

0.664 

0.709 

0.977 

0.952 

1 

4 

2 

-1.7 

-0.007 

0.004 

0.084 

1 

0 

0.866 

1.008 

1.03 

0.883 

0.663 

1.045 

1.082 

1.03 

0.916 

0.711 

0.754 

0.739 

0.848 

1.084 

0.871 

3 

-3 

-1 

2.1 

0 

0.027 

0.055 

0 

2 

0.829 

0.829 

0.7 

0.711 

0.746 

0.7 

0.647 

0.618 

0.778 

0.987 

0.905 

1.034 

0.885 

1.086 

0.946 

2 

4 

0 

3.7 

-0.023 

0.007 

0.071 

1 

-2 

0.934 

0.963 

0.944 

0.913 

0.827 

0.842 

0.653 

0.815 

0.62 

0.877 

1.086 

1.002 

0.875 

0.973 

1.059 

2 

-3 

3 

1.5 

-0.016 

0.041 

0.046 

2 

-2 

0.657 

0.604 

0.659 

0.807 

1.065 

0.895 

0.905 

0.993 

1.004 

0.944 

0.92 

0.989 

1.077 

1.022 

1.098 

2 

3 

3 

2.1 

0.028 

0.043 

0.087 

0 

2 

0.879 

0.993 

0.957 

1.057 

0.913 

0.954 

0.975 

1.02 

0.852 

1.008 

1.098 

0.948 

0.969 

0.938 

0.823 

0 

3 

1 

1.7 

-0.006 

0.04 

0.1 

2 

-1 

0.719 

0.793 

0.631 

0.692 

0.77 

0.797 

0.823 

0.629 

0.618 

0.686 

0.739 

0.721 

0.746 

0.788 

0.696 

-1 

-1 

2 

3.7 

0.008 

0.003 

0.096 

1 

0 

0.913 

0.971 

1.1 

1.043 

0.856 

0.612 

0.631 

0.668 

0.602 

0.776 

0.889 

0.719 

0.807 

0.86 

0.725 

-1 

1 

-1 

1.4 

0.044 

0.031 

0.091 

2 

1 

0.702 

0.723 

0.664 

0.672 

1.069 

1.016 

0.883 

1.065 

1.022 

0.721 

0.641 

0.676 

0.618 

0.754 

0.608 

0 

-3 

1 

1.8 

0.01 

0.025 

0.082 

2 

-1 

1.069 

0.991 

1.082 

1.077 

0.922 

0.891 

0.995 

1.1 

0.989 

0.608 

0.67 

0.705 

0.823 

0.717 

0.797 

0 

0 

1 

0.4 

0.039 

0.029 

0.063 

1 

-2 

0.643 

0.627 

0.76 

0.639 

0.852 

0.68 

0.674 

0.657 

0.739 

1.065 

0.928 

1.022 

1.03 

0.746 

0.77 

0 

-1 

3 

-1 

0.029 

0.041 

0.06 

2 

1 

1.02 

0.981 

1.02 

0.954 

1.057 

0.813 

0.791 

0.821 

0.786 

1.045 

1.036 

1.053 

1.09 

0.885 

0.616 

-2 

1 

1 

-1.3 

-0.023 

0.002 

0.065 

0 

1 

0.602 

0.735 

0.754 

0.76 

0.672 

1.055 

1.057 

0.93 

1.09 

0.998 

0.844 

1.08 

1.008 

0.752 

0.801 

0 

-4 

0 

0.4 

0.039 

0.024 

0.055 

1 

0 

0.832 

0.866 

1.004 

0.926 

0.813 

1.053 

0.965 

1.016 

0.907 

0.94 

1.024 

1.079 

1.061 

0.713 

0.735 

-2 

0 

3 

1.8 

-0.042 

0.029 

0.061 

2 

-1 

1.073 

0.715 

0.829 

0.815 

0.692 

0.928 

0.803 

0.748 

0.77 

0.909 

:.s=- 

0.635 

0.713 

1.034 

i 

0 

-3 

2 

2.3 

-0.094 

0.011 

0.035 

0 

2 

0.801 

0.942 

0.979 

1.075 

0.666 

0.879 

0.821 

0.655 

0.727 

1.012 

0.848 

0.737 

0.809 

0.858 

0.987 

-1 

3 

1 

3.8 

-0.034 

0 

0.096 

0 

2 

0.922 

0.705 

0.705 

0.801 

0.924 

0.651 

0.985 

0.86 

0.864 

1.036 

0.815 

0.606 

0.862 

1.09 

0.969 

-2 

-3 

-1 

2.4 

-0.045 

0.017 

0.065 

0 

0 

0.819 

0.846 

1.09 

0.918 

0.959 

0.817 

0.899 

1 

1.018 

0.907 

0.731 

0.756 

0.741 

0.993 

0.971 

-1 

4 

2 

-1.6 

-0.039 

0.03 

0.047 

1 

-1 

0.983 

0.704 

0.621 

0.836 

1.006 

0.934 

0.729 

0.639 

0.702 

0.819 

0.952 

1.02 

0.93 

0.94 

1.039 

-2 

-2 

2 

-1.2 

-0.076 

0.03 

0.098 

0 

1 

0.776 

0.989 

0.936 

0.93 

0.901 

1.034 

0.811 

0.772 

0.754 

0.821 

0.858 

1.084 

0.907 

1.032 

1.073 

-1 

1 

-1 

0.9 

-0.082 

0.003 

0.064 

0 

1 

0.883 

0.733 

0.623 

0.674 

0.655 

0.752 

1.02 

1.079 

0.871 

0.661 

1.02 

0.871 

0.883 

0.825 

1.053 

0 

-2 

0 

-1.6 

-0.04 

0.022 

0.091 

1 

-2 

0.606 

0.864 

0.987 

0.889 

0.688 

0.784 

0.954 

0.965 

0.879 

0.836 

0.993 

0.926 

1.055 

1.012 

1.1 
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-1 

0 

2 

2.7 

0.035 

-0.013 

0.046 

1 

-1 

0.854 

0.926 

0.661 

0.764 

0.766 

0.768 

1.014 

0.676 

0.655 

0.758 

1.028 

0.661 

0.803 

0.92 

0.879 

0 

-2 

2 

1.7 

-0.02 

-0.022 

0.078 

1 

1 

1.053 

0.721 

0.905 

0.914 

0.838 

0.686 

1.043 

0.692 

0.676 

0.842 

0.897 

0.75 

0.756 

1.08 

0.756 

-1 

2 

1 

1 

0.022 

-0.046 

0.081 

1 

1 

0.721 

0.961 

0.678 

0.649 

0.946 

0.864 

0.793 

0.983 

0.969 

0.606 

1.073 

0.651 

0.69 

1.069 

0.707 

-1 

-2 

0 

2 

0.012 

-0.023 

0.067 

2 

0 

1.002 

0.61 

1.028 

1.004 

0.987 

1.024 

0.743 

0.955 

0.883 

0.694 

1.08 

0.653 

0.614 

0.932 

0.664 

0 

0 

2 

-0.8 

0.019 

-0.043 

0.098 

2 

-1 

0.614 

0.998 

0.655 

0.684 

1.063 

0.748 

1.059 

0.645 

0.805 

0.993 

0.674 

0.946 

1.067 

0.914 

0.782 

0 

-1 

1 

-0.8 

0.036 

-0.036 

0.08 

1 

1 

0.971 

0.621 

1.075 

1.092 

0.842 

0.737 

0.809 

0.721 

0.723 

0.963 

0.639 

1.092 

0.973 

0.893 

0.788 

0 

2 

1 

0.8 

-0.022 

-0.002 

0.067 

1 

1 

0.709 

1.047 

0.686 

0.778 

0.748 

1.092 

0.651 

1.039 

1.028 

0.829 

0.737 

1.008 

0.963 

1.063 

0.741 

-1 

-1 

0 

-0.3 

0.011 

-0.043 

0.072 

1 

0 

0.842 

0.852 

0.967 

0.971 

0.727 

0.94 

0.6 

1.086 

1.069 

1.002 

0.774 

0.938 

0.922 

0.868 

0.83 

-1 

0 

2 

3.4 

-0.068 

-0.011 

0.03 

2 

-2 

0.911 

1.016 

0.746 

0.627 

0.805 

0.983 

0.983 

0.76 

0.834 

1 

0.854 

0.815 

0.659 

0.676 

1.086 

0 

-2 

3 

1.2 

-0.041 

-0.046 

0.053 

2 

1 

0.707 

0.727 

0.842 

1.08 

0.75 

0.856 

0.858 

0.797 

0.664 

0.817 

1.038 

0.832 

0.62 

0.87 

0.977 

-2 

2 

1 

2.9 

-0.026 

-0.006 

0.041 

1 

1 

0.936 

0.875 

0.745 

0.829 

0.877 

0.811 

0.664 

0.977 

1.061 

0.954 

1.077 

0.743 

0.827 

0.69 

1.028 

-1 

-3 

0 

25 

-0.05 

-0.013 

0.054 

0 

0 

0.862 

0.811 

0.883 

1.002 

1.09 

0.721 

0.752 

0.881 

1.055 

0.996 

0.891 

0.674 

0.684 

0.625 

1.067 

-1 

3 

3 

-0.8 

-0.07 

-0.009 

0.075 

1 

0 

0.975 

0.987 

0.715 

0.789 

1.024 

1.002 

0.868 

0.864 

0.637 

0.731 

0.823 

0.954 

0.938 

0.733 

0.981 

-1 

-2 

1 

-1.4 

-0.07 

-0.034 

0.063 

0 

2 

0.678 

0.815 

0.998 

0.969 

0.92 

0.793 

1.045 

0.766 

0.625 

0.832 

0.86 

0.916 

0.967 

0.623 

1.006 

0 

3 

0 

0.9 

-0.029 

-0.038 

0.051 

2 

1 

0.944 

0.897 

0.805 

0.629 

0.661 

0.692 

0.676 

0.975 

0.918 

0.778 

0.713 

0.94 

1.059 

0.704 

1.018 

-1 

0 

0 

-0.5 

-0.058 

-0.01 

0.031 

2 

-1 

0.811 

0.67 

0.918 

1.049 

0.709 

0.633 

0.813 

1.057 

1 

0.668 

0.797 

0.881 

1.069 

0.758 

0.92 

0 

1 

2 

1.2 

-0.004 

0.05 

-0.035 

1 

-1 

0.67 

0.614 

0.891 

0.813 

0.67 

0.69 

0.694 

0.95 

0.608 

0.676 

0.705 

0.955 

0.702 

0.657 

1.034 

0 

0 

2 

2.1 

0.028 

0.047 

0.02 

0 

2 

0.907 

1.082 

0.682 

0.975 

0.789 

0.713 

0.627 

1.077 

0.815 

0.623 

0.715 

0.846 

0.819 

0.737 

0.844 

-1 

3 

0 

1.6 

0.031 

0.044 

-0.022 

0 

0 

0.705 

0.688 

1.008 

0.645 

0.875 

1.022 

1.096 

0.778 

0.873 

0.723 

0.733 

1.016 

0.784 

0.629 

1.024 

-1 

-2 

1 

2.2 

0.017 

0.038 

-0.012 

2 

0 

1.026 

1.075 

0.707 

0.895 

0.871 

0.973 

1.094 

0.737 

0.983 

0.621 

0.633 

0.952 

0.754 

0.813 

1.03 

0 

1 

1 

-1 

0.048 

0.025 

-0.019 

1 

0 

0.623 

0.754 

1.073 

0.827 

0.834 

0.629 

0.698 

0.959 

0.719 

0.961 

1.047 

0.668 

1.032 

0.692 

0.942 

0 

-2 

2 

-0.5 

0.03 

0.023 

-0.036 

0 

0 

1.084 

1.024 

0.717 

1.022 

0.881 

0.674 

0.721 

0.957 

0.819 

0.981 

1.057 

0.67 

0.95 

0.635 

0.838 

0 

0 

1 

0 

0.017 

0.027 

0.022 

1 

1 

0.664 

0.756 

0.996 

0.762 

0.696 

1.03 

0.942 

0.616 

0.862 

1.08 

1.088 

0.801 

0.836 

0.805 

0.889 

0 

-1 

0 

0.8 

0.022 

0.02 

-0.011 

1 

0 

0.977 

1.032 

0.848 

0.967 

0.657 

1.082 

0.991 

0.756 

1.053 

0.936 

1.045 

0.623 

0.993 

0.7 

0.998 

-1 

2 

3 

1.8 

-0.03 

0.024 

-0.018 

1 

-1 

0.95 

0.776 

0.913 

0.776 

0.653 

0.918 

0.782 

0.954 

0.797 

0.848 

0.604 

0.977 

0.86 

1.002 

0.819 

-1 

-3 

2 

3.1 

-0.053 

0.007 

0.009 

0 

2 

0.848 

0.922 

0.711 

0.944 

0.684 

0.955 

0.62 

0.871 

0.606 

0.875 

0.684 

0.879 

0.821 

1.041 

0.84 

-1 

4 

0 

3.2 

-0.041 

0.048 

0.024 

2 

2 

0.928 

0.694 

0.948 

0.809 

1 

0.756 

0.967 

0.832 

0.875 

0.985 

0.686 

0.973 

0.621 

1.045 

0.637 

-2 

-3 

0 

2.7 

-0.054 

0.027 

0 

1 

-1 

0.809 

1.088 

0.862 

1.088 

0.996 

0.868 

0.946 

0.776 

0.911 

1.053 

0.69 

0.936 

0.832 

0.959 

0.686 

-1 

3 

2 

-1.9 

-0.075 

0.014 

-0.02 

0 

-1 

0.954 

0.623 

1.016 

0.618 

0.957 

0.834 

0.602 

0.948 

0.84 

0.752 

1.041 

0.786 

0.889 

1.039 

0.733 

-1 

-2 

2 

-1.2 

-0.063 

0.049 

-0.02 

2 

1 

0.635 

0.936 

0.725 

0.84 

1.049 

0.905 

0.776 

0.914 

0.811 

0.786 

1.053 

0.827 

1.075 

1.1 

0.698 

-1 

1 

0 

-0.7 

-0.051 

0.026 

0.021 

0 

1 

0.83 

0.663 

0.973 

0.834 

0.623 

0.704 

1.067 

0.696 

1.065 

0.813 

0.955 

0.789 

0.871 

1.038 

0.795 

0 

-3 

0 

-0.9 

-0.031 

0.044 

-0.008 

0 

-1 

0.731 

1.041 

0.86 

0.916 

0.621 

0.649 

0.911 

0.733 

0.856 

0.87 

0.914 

0.633 

0.942 

1.073 

0.678 
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-2 

2 

2 

2.4 

0.05 

-0.014 

0.006 

0 

-2 

0.637 

1.036 

0.959 

0.653 

0.733 

0.639 

1.039 

0.895 

0.78 

0.748 

0.981 

1.01 

0.655 

0.881 

0.963 

-1 

-4 

3 

1.4 

0.046 

-0.035 

-0.008 

2 

0 

1.1 

0.743 

0.772 

0.846 

0.604 

0.623 

1.008 

1.096 

0.725 

0.674 

0.977 

0.907 

0.68 

1.014 

1.071 

-2 

1 

0 

3.9 

0.041 

-0.037 

-0.044 

1 

2 

0.684 

1.01 

0.938 

0.799 

0.914 

1.057 

0.739 

0.702 

1.067 

0.666 

1.018 

1.041 

0.739 

1.047 

0.995 

0 

-4 

-1 

2.6 

0.032 

-0.028 

-0.01 

1 

-1 

1.092 

0.682 

0.606 

1.045 

0.916 

LI 

0.748 

0.807 

0.893 

0.788 

1.012 

LI 

0.846 

1.02 

0.885 

-1 

4 

2 

0.1 

0.013 

-0.048 

-0.039 

0 

0 

0.741 

1.061 

1.026 

0.754 

1.092 

0.688 

0.998 

1.038 

0.709 

0.955 

0.717 

0.713 

0.92 

0.965 

0.907 

-2 

-2 

1 

-1.9 

0.01 

-0.037 

-0.029 

1 

1 

0.961 

0.655 

0.612 

0.868 

0.94 

0.663 

1.01 

0.858 

0.694 

1.086 

0.629 

0.639 

1.049 

0.922 

0.868 
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APPENDIX  F.  MSM  CLASS  ORGANIZATION 


Source  Package 


data.hist  S  model.structure 

S  GainData.java 

|S  GAR.java 

S  LossData.java 

|S  GarTre.java 

[S  Promo tionData  .java 

-|S  Marine.java 

@@  data. user 

[S  Mos.java 

Bl  Data.java 

|S  Network. java 

[S  DataByFy.java 

S  Node  .java 

S  DataByFyGrade  .java 

|S  SNode.java 

[S  DataByFyMos.java  00  orade 

|S  DataSet. java 

[5]  OradeData  .java 

Si  DateFromSeq.java 

|S  OradeDateFromSeq.java 

■  S  Distributions  .java 

S  OradeGains  .java 

|S  ForceControls  .java 

S  OradelnvSeq.java 

IS  InvSeq.java 

-  |S  OradeLosses  .java 

|S  Inventory  .java 

-  |S  OradeODSEInventory  .java 

|S  PayData.java 

S  OradePromotions.java 

|S  RunArgs.java 

|S  OradeSequence  .java 

|S  Sequence  .java 

S  OradeTFD Win ventory  .java 

£ 

l^main 

S  OradeTargetDistros.java 

|S  InputPrep.java  @0  output 

|flft  InventoryDataTest.java 

|S  AnnualStatsAggrega  tor  .java 

fe  Main  .java 

|S  MonthlyStatsAggrega  tor  .java 

-|S  Simulation  .java 

|S  ResultsManager.java 

b 

3  main.gui 

S  Writer  .java 

|S  GenericAction.java  test 

|S  MSMMenubar.java 

IS  TestCreatelnputs.java 

S  MSMToolBar.java 

|S  TestPOI.java 

IS  MainGui.java  FEUutils 

|S  WindowCloser.java 

|S  ExcelFileType.java 

@@  model. process 

S  T abNotFoundException  .java 

|S  Accessions  .java 

jS  UnsupportedFileTypeException.java 

IS  Classification  .java 

|S  Utils.java 

S  LatMove.java 
|S  LatMovePromotions.java 
Stosses  .java 

S  Process.java 

S)  Promote. java 

IS  RegularPromotions.java 
is  Retention  .java 

IS  Version.java 

Test  Package 

ffi  utils 

[• ■••• S^xcelFileTypeTest.java 
l-  [fiJ^VersionTest.java 


Libraries 

^  Libraries 

©  B  Apache  POI  -  poi-3. 11-20 14 1221. jar 

5  B  Apache  POI  -  poi-ooxml-3. 11-20 14 1221. jar 

©  B  Apache  POI  -  poi-ooxml-schemas-3. 11-20 14 1221. jar 

©  B  Apache  POI  -  xmlbeans -2. 6.0. jar 

©  B  Apache  Commons  Math  -  commons-math3-3. 5. jar 

(j)  B  MersenneTwisterFast  -  MersenneTwisterFast.jar 

GjD  B  ojdbc -ojdbcl4.jar 

l|)  B  JUnit4. 10  -  junit-4.10.jar 

©  Sj  JDK  1.8  (Default) 


Figure  53.  MSM  Class  Organization.  Adapted  from  Garrick  (201 4). 
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APPENDIX  G.  VALIDATION  RESULT  TABLES 


Table  1 1 .  NEAS  Losses  Relative  Error 


lyr-Forecast 

2yr-Forecast 

3yr-Forecast 

4yr-Forecast 

5yr-Forecast 

2004 

-54% 

-49% 

-45% 

-43% 

-47% 

2005 

-53% 

-47% 

-43% 

-39% 

-47% 

2006 

-59% 

-56% 

-51% 

-51% 

-51% 

2007 

-52% 

-49% 

-48% 

-41% 

-44% 

2008 

-53% 

-54% 

-46% 

-42% 

-47% 

2009 

-51% 

-46% 

-40% 

-40% 

-46% 

2010 

-43% 

-35% 

-32% 

-35% 

-44% 

2011 

-38% 

-35% 

-34% 

-35% 

-49% 

2012 

-33% 

-33% 

-31% 

-43% 

2013 

-33% 

-32% 

-41% 

2014 

-33% 

-44% 

Table  12.  Retired  Losses  Relative  Error 


lyr-Forecast 

2yr-Forecast 

3yr-Forecast 

4yr-Forecast 

5yr-Forecast 

2004 

-25% 

-39% 

-23% 

-43% 

-20% 

2005 

-18% 

-29% 

-13% 

-39% 

-32% 

-17% 

-22% 

-49% 

-22% 

-26% 

2008 

-22% 

-50% 

-25% 

-41% 

-49% 

2009 

-16% 

-11% 

-11% 

-53% 

-31% 

2010 

18% 

-8% 

-26% 

-29% 

-58% 

2011 

-6% 

-44% 

-20% 

-60% 

-79% 

2012 

-22% 

-26% 

-47% 

-81% 

2013 

-5% 

-48% 

-71% 

2014 

-33% 

-72% 

Table  13.  Other  Losses  Relative  Error 


lyr-Forecast 

2yr-Forecast 

3yr-Forecast 

4yr-Forecast 

5yr-Forecast 

2004 

-29% 

-11% 

-19% 

-12% 

16% 

2005 

-8% 

-20% 

1% 

47% 

20% 

2006 

-40% 

-23% 

28% 

8% 

20% 

2007 

-6% 

74% 

82% 

1 1 2% 

1 23% 

2008 

-22% 

18% 

49% 

97% 

73% 

2009 

-21% 

-6% 

27% 

25% 

68% 

2010 

3% 

36% 

62% 

1 1 6% 

62% 

2011 

7% 

11% 

74% 

43% 

5% 

2012 

-5% 

31% 

23% 

-9% 

2013 

6% 

-3% 

-21% 

2014 

6% 

-23% 
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Table  14.  EAS  Losses  Relative  Error 


lyr-Forecast 

2yr-Forecast 

3yr-Forecast 

4yr-Forecast 

5yr-Forecast 

2004 

-6% 

-3% 

7% 

11% 

-49% 

2005 

-4% 

-2% 

14% 

29% 

-53% 

2006 

-7% 

4% 

32% 

24% 

-55% 

2007 

-2% 

19% 

23% 

11% 

-58% 

2008 

-8% 

-6% 

-5% 

-5% 

-57% 

2009 

-15% 

-19% 

-9% 

-10% 

-50% 

2010 

-16% 

-12% 

-4% 

-6% 

-56% 

2011 

-10% 

-10% 

-1% 

-17% 

-28% 

2012 

-4% 

-3% 

-10% 

1% 

2013 

-6% 

-18% 

2% 

2014 

-22% 

-12% 

Table  15.  Aggregated  Gains  Relative  Error 


lyr-Forecast 

2yr-Forecast 

3yr-Forecast 

4yr-Forecast 

5yr-Forecast 

2004 

2% 

-41% 

-34% 

-42% 

-38% 

2005 

-8% 

-40% 

-46% 

-38% 

-26% 

2006 

1% 

-52% 

-38% 

-19% 

49% 

mm 

-28% 

-52% 

46% 

63% 

mm 

13% 

1 20% 

243% 

2009 

16% 

34% 

77% 

1 97% 

253% 

2010 

66% 

41% 

1 53% 

209% 

237% 

2011 

14% 

39% 

87% 

1 07% 

43% 

2012 

46% 

59% 

29% 

2013 

16% 

29% 

2014 

19% 

-23% 

Table  16.  Accession  Gains  Relative  Error 


lyr-Forecast 

2yr-Forecast 

3yr-Forecast 

4yr-Forecast 

5yr-Forecast 

2004 

-100% 

-28% 

14% 

-4% 

-29% 

2005 

-100% 

-5% 

-1% 

10% 

-7% 

2006 

-100% 

-20% 

8% 

36% 

-32% 

2007 

-100% 

0% 

37% 

18% 

-10% 

2008 

-100% 

4% 

2% 

29% 

-16% 

2009 

-100% 

-17% 

27% 

24% 

-64% 

2010 

-100% 

26% 

35% 

-24% 

-49% 

2011 

-50% 

30% 

-1 8% 

-5% 

-33% 

2012 

-5% 

-21% 

3% 

-9% 

2013 

-19% 

-6% 

-7% 

2014 

-24% 

-19% 
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Table  17.  Promotions  to  E-2  Relative  Error 


lyr-Forecast 

2yr-Forecast 

3yr-Forecast 

4yr-Forecast 

5yr-Forecast 

2004 

-40% 

-68% 

-20% 

-5% 

-30% 

2005 

-44% 

-60% 

-11% 

-12% 

-19% 

2006 

-42% 

-62% 

-1 7% 

-1% 

-2% 

2007 

-43% 

-55% 

-4% 

21% 

-2% 

2008 

-44% 

-63% 

-3% 

9% 

-1% 

2009 

-41% 

-64% 

1% 

19% 

-26% 

2010 

-44% 

-41% 

21% 

-4% 

-37% 

2011 

-21% 

-15% 

-2% 

-14% 

-18% 

2012 

3% 

-18% 

-10% 

1% 

2013 

-5% 

-11% 

-1% 

2014 

-6% 

-16% 

Table  18.  Promotions  to  E-3  Relative  Error 


lyr-Forecast 

2yr-Forecast 

3yr-Forecast 

4yr-Forecast 

5yr-Forecast 

2004 

-11% 

-69% 

-42% 

-4% 

-13% 

2005 

-10% 

-69% 

-28% 

-8% 

-12% 

2006 

-10% 

-67% 

-32% 

-8% 

1% 

-67% 

5% 

12% 

mm 

-1% 

2% 

2009 

-10% 

-68% 

-31% 

5% 

3% 

2010 

-13% 

-64% 

-10% 

12% 

-29% 

2011 

-5% 

-37% 

-2% 

-21% 

-19% 

2012 

-1% 

-10% 

-26% 

-9% 

2013 

-1% 

-21% 

-12% 

2014 

-4% 

-20% 

Table  19.  Promotions  to  E-4  Relative  Error 


lyr-Forecast 

2yr-Forecast 

3yr-Forecast 

4yr-Forecast 

5yr-Forecast 

2004 

-61% 

-46% 

-18% 

-27% 

-20% 

2005 

-67% 

-20% 

-24% 

21% 

-38% 

2006 

-62% 

-33% 

16% 

7% 

-50% 

2007 

-66% 

9% 

4% 

-6% 

-41% 

2008 

-67% 

-25% 

-36% 

-24% 

-73% 

2009 

-60% 

-40% 

-28% 

-48% 

-55% 

2010 

-63% 

-24% 

-42% 

-34% 

-53% 

2011 

-40% 

-35% 

-24% 

-29% 

-41% 

2012 

-41% 

-16% 

-19% 

-18% 

2013 

-23% 

-20% 

-11% 

2014 

-25% 

-22% 
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Table  20.  Promotions  to  E-5  Relative  Error 


lyr-Forecast 

2yr-Forecast 

3yr-Forecast 

4yr-Forecast 

5yr-Forecast 

2004 

-94% 

-65% 

-42% 

-51% 

-15% 

2005 

-93% 

-51% 

-44% 

-7% 

-9% 

2006 

-88% 

-58% 

-6% 

-3% 

-23% 

2007 

-89% 

-20% 

1% 

-9% 

-5% 

2008 

-82% 

-39% 

-49% 

-24% 

-68% 

2009 

-68% 

-50% 

-16% 

-59% 

-44% 

2010 

-84% 

-27% 

-57% 

-42% 

-70% 

2011 

-49% 

-50% 

-20% 

-51% 

-59% 

2012 

-69% 

-25% 

-42% 

-49% 

2013 

-31% 

-37% 

-30% 

2014 

-31% 

-39% 

Table  21 .  Promotions  to  E-6  Relative  Error 


lyr-Forecast 

2yr-Forecast 

3yr-Forecast 

4yr-Forecast 

5yr-Forecast 

2004 

-96% 

-54% 

-43% 

-36% 

-2% 

2005 

-93% 

-56% 

-18% 

-3% 

-5% 

2006 

-77% 

-32% 

10% 

-4% 

-40% 

mm 

-73% 

-4% 

6% 

-42% 

mm 

-44% 

-11% 

-8% 

-49% 

2009 

-29% 

-43% 

3% 

-52% 

-9% 

2010 

-44% 

4% 

-32% 

1% 

-74% 

2011 

-29% 

-41% 

17% 

-65% 

-47% 

2012 

-43% 

14% 

-48% 

2013 

19% 

-50% 

-21% 

2014 

-8% 

-19% 

Table  22.  Promotions  to  E-7  Relative  Error 


lyr-Forecast 

2yr-Forecast 

3yr-Forecast 

4yr-Forecast 

5yr-Forecast 

2004 

-91% 

-66% 

-43% 

-37% 

21% 

2005 

-88% 

-65% 

-11% 

12% 

-19% 

2006 

-79% 

-39% 

31% 

-30% 

-39% 

2007 

-78% 

7% 

-1 3% 

-48% 

11% 

2008 

-50% 

-31% 

-42% 

-1% 

-41% 

2009 

-40% 

-30% 

24% 

-43% 

-16% 

2010 

-40% 

30% 

-1 0% 

-6% 

-55% 

2011 

-24% 

-34% 

0% 

-59% 

-42% 

2012 

-31% 

-6% 

-30% 

-42% 

2013 

11% 

-35% 

-1 7% 

2014 

-23% 

-30% 
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Table  23.  Promotions  to  E-8  Relative  Error 


lyr-Forecast 

2yr-Forecast 

3yr-Forecast 

4yr-Forecast 

5yr-Forecast 

2004 

-94% 

-96% 

-40% 

-57% 

67% 

2005 

-96% 

-69% 

-35% 

45% 

6% 

2006 

-90% 

-61% 

68% 

-16% 

-27% 

2007 

-90% 

36% 

22% 

-44% 

20% 

2008 

-42% 

-16% 

-32% 

-4% 

-32% 

2009 

-25% 

-10% 

42% 

-29% 

-6% 

2010 

-9% 

48% 

8% 

0% 

-20% 

2011 

-5% 

-26% 

4% 

-27% 

-30% 

2012 

-15% 

-2% 

9% 

-37% 

2013 

16% 

4% 

5% 

2014 

-26% 

-23% 

Table  24.  Promotions  to  E-9  Relative  Error 


lyr-Forecast 

2yr-Forecast 

3yr-Forecast 

4yr-Forecast 

5yr-Forecast 

2004 

-93% 

-97% 

-42% 

-47% 

1 25% 

2005 

-96% 

-74% 

-29% 

101% 

11% 

2006 

-92% 

-72% 

1 34% 

-21% 

13% 

mm 

74% 

16% 

-21% 

33% 

mm 

-18% 

-29% 

-1% 

-9% 

-10% 

2009 

-6% 

53% 

45% 

-6% 

21% 

2010 

24% 

38% 

27% 

3% 

1% 

2011 

7% 

-15% 

18% 

-26% 

-5% 

2012 

5% 

37% 

23% 

-22% 

2013 

62% 

5% 

15% 

2014 

-19% 

-6% 

Table  25.  End  Strength  Relative  Error 


lyr-Forecast 

2yr-Forecast 

3yr-Forecast 

4yr-Forecast 

5yr-Forecast 

2004 

-15% 

-20% 

-18% 

-19% 

-19% 

2005 

-17% 

-18% 

-19% 

-19% 

-15% 

2006 

-16% 

-19% 

-19% 

-15% 

-12% 

2007 

-19% 

-19% 

-15% 

-12% 

-6% 

2008 

-17% 

-15% 

-12% 

-6% 

0% 

2009 

-13% 

-12% 

-6% 

0% 

-2% 

2010 

-12% 

-6% 

0% 

-2% 

0% 

2011 

-6% 

0% 

-2% 

0% 

0% 

2012 

0% 

-2% 

0% 

2013 

-2% 

0% 

0% 

2014 

0% 

0% 
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Table  26.  Retention  Zone-A  Relative  Error 


lyr-Forecast 

2yr-Forecast 

3yr-Forecast 

4yr-Forecast 

5yr-Forecast 

2004 

3% 

-6% 

-6% 

-49% 

-90% 

2005 

-1% 

-4% 

-51% 

-40% 

-84% 

2006 

8% 

-48% 

-39% 

-28% 

-80% 

2007 

-42% 

-36% 

-32% 

-23% 

-68% 

2008 

4% 

-3% 

3% 

38% 

-43% 

2009 

11% 

11% 

84% 

28% 

-53% 

2010 

-2% 

53% 

24% 

-7% 

-41% 

2011 

55% 

19% 

-1% 

11% 

11% 

2012 

-2% 

-18% 

-2% 

8% 

2013 

-10% 

6% 

15% 

2014 

21% 

33% 

Table  27.  Retention  Zone-B  Relative  Error 


lyr-Forecast 

2yr-Forecast 

3yr-Forecast 

4yr-Forecast 

5yr-Forecast 

2004 

5% 

4% 

31% 

-3% 

10% 

2005 

-4% 

-10% 

-8% 

13% 

11% 

2006 

-8% 

-29% 

23% 

38% 

57% 

mm 

-29% 

-9% 

46% 

83% 

mm 

9% 

34% 

117% 

1 00% 

97% 

18% 

91% 

121% 

1 37% 

1 68% 

2010 

59% 

77% 

131% 

1 60% 

1 28% 

2011 

37% 

74% 

1 36% 

1 1 7% 

80% 

2012 

41% 

94% 

113% 

78% 

2013 

55% 

69% 

83% 

2014 

44% 

51% 

Table  28.  Retention  Zone-C  Relative  Error 


lyr-Forecast 

2yr-Forecast 

3yr-Forecast 

4yr-Forecast 

5yr-Forecast 

2004 

19% 

34% 

43% 

-3% 

9% 

2005 

22% 

27% 

1% 

19% 

31% 

2006 

17% 

-4% 

35% 

58% 

50% 

2007 

-12% 

20% 

69% 

69% 

22% 

2008 

16% 

63% 

95% 

64% 

35% 

2009 

28% 

72% 

60% 

41% 

55% 

2010 

36% 

44% 

42% 

67% 

110% 

2011 

10% 

16% 

57% 

86% 

64% 

2012 

-5% 

48% 

114% 

86% 

2013 

36% 

1 23% 

97% 

2014 

92% 

89% 
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Table  29.  Retention  Zone-D  Relative  Error 


lyr-Forecast 

2yr-Forecast 

3yr-Forecast 

4yr-Forecast 

5yr-Forecast 

2004 

4% 

18% 

42% 

-11% 

29% 

2005 

8% 

33% 

-4% 

33% 

77% 

2006 

18% 

-7% 

51% 

98% 

18% 

2007 

-17% 

45% 

116% 

24% 

34% 

2008 

31% 

1 07% 

43% 

51% 

31% 

2009 

65% 

38% 

55% 

34% 

18% 

2010 

12% 

48% 

37% 

25% 

75% 

2011 

24% 

28% 

38% 

75% 

75% 

2012 

8% 

29% 

96% 

93% 

2013 

12% 

85% 

1 23% 

2014 

51% 

101% 

Table  30.  Retention  Zone-E  Relative  Error 


lyr-Forecast 

2yr-Forecast 

3yr-Forecast 

4yr-Forecast 

5yr-Forecast 

2004 

1 53% 

1 83% 

1 95% 

75% 

1 87% 

2005 

1 78% 

1 88% 

101% 

214% 

233% 

2006 

1 86% 

116% 

295% 

309% 

1 38% 

■MiTiUI 

1 84% 

1 54% 

2008 

451% 

2009 

452% 

303% 

225% 

209% 

1 92% 

2010 

268% 

243% 

227% 

21 3% 

261% 

2011 

240% 

244% 

299% 

31 3% 

256% 

2012 

227% 

287% 

417% 

31 2% 

2013 

269% 

407% 

423% 

2014 

370% 

415% 

Table  31 .  Retention  Zone-Senior  Enlisted  Relative  Error 


lyr-Forecast 

2yr-Forecast 

3yr-Forecast 

4yr-Forecast 

5yr-Forecast 

2004 

1 33% 

112% 

1 27% 

-1% 

398% 

2005 

121% 

90% 

22% 

1 73% 

116% 

2006 

1 06% 

22% 

302% 

31% 

43% 

2007 

26% 

300% 

92% 

-4% 

83% 

2008 

392% 

161% 

89% 

98% 

101% 

2009 

82% 

52% 

67% 

12% 

53% 

2010 

45% 

58% 

36% 

15% 

44% 

2011 

114% 

91% 

1 08% 

60% 

87% 

2012 

86% 

1 05% 

1 08% 

46% 

2013 

76% 

101% 

79% 

2014 

1 28% 

1 24% 
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